2021-01-15T18:25:55.542 INFO:root:teuthology version: 1.0.0-04fddc54 2021-01-15T18:25:55.544 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643 --lock --block --name teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi --description rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} -- /tmp/teuthology-worker.w5nc40w1.tmp 2021-01-15T18:25:55.576 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T18:25:55.646 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643 branch: master description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} email: ceph-qa@ceph.io first_in_suite: false job_id: '5788643' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.3' overrides: admin_socket: branch: master ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: master sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 owner: scheduled_teuthology@teuthology priority: 80 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 sleep_before_teardown: 0 suite: rados suite_branch: master suite_path: /home/teuthworker/src/git.ceph.com_ceph_master/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - ec_lost_unfound: null teuthology_branch: master timestamp: 2021-01-15_07:01:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6159 2021-01-15T18:25:55.647 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_master/qa; will attempt to use it 2021-01-15T18:25:55.648 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks 2021-01-15T18:25:55.648 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-01-15T18:25:55.650 INFO:teuthology.task.internal:Checking packages... 2021-01-15T18:25:55.671 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '892c51dd3c5f7108e766bea30cd5e0d801b0abd3' 2021-01-15T18:25:55.672 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-01-15T18:25:55.672 INFO:teuthology.packaging:ref: None 2021-01-15T18:25:55.672 INFO:teuthology.packaging:tag: None 2021-01-15T18:25:55.673 INFO:teuthology.packaging:branch: master 2021-01-15T18:25:55.673 INFO:teuthology.packaging:sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T18:25:55.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2021-01-15T18:25:56.292 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-9044.gb9095837 2021-01-15T18:25:56.292 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-01-15T18:25:56.300 INFO:teuthology.task.internal:no buildpackages task found 2021-01-15T18:25:56.300 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2021-01-15T18:25:56.448 INFO:teuthology.task.internal.lock_machines:Locking machines... 2021-01-15T18:25:56.449 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T18:25:56.508 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643'} 2021-01-15T18:25:56.525 DEBUG:teuthology.lock.ops:locked ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:25:56.553 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60b7af50d0>, signals=[15]) 2021-01-15T18:25:56.761 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of rhel 8.3 2021-01-15T18:25:57.080 INFO:teuthology.orchestra.console:Power off smithi148 2021-01-15T18:25:57.081 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-15T18:25:57.110 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:01.141 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:05.254 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:05.464 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-01-15T18:26:05.569 INFO:teuthology.orchestra.console:Power on smithi148 2021-01-15T18:26:05.569 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-15T18:26:05.591 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:09.611 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:13.722 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T18:26:13.933 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2021-01-15T18:26:14.034 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2021-01-15T18:28:45.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:28:54.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:03.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:12.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:21.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:30.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:39.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:29:48.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:30:54.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:30:54.895 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-01-15T18:30:54.897 DEBUG:teuthology.orchestra.run.smithi148:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2021-01-15T18:30:55.205 INFO:teuthology.provision.fog.smithi148:Node is ready 2021-01-15T18:30:55.206 DEBUG:teuthology.orchestra.run.smithi148:> hostname 2021-01-15T18:30:55.220 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2021-01-15T18:30:55.221 DEBUG:teuthology.orchestra.run.smithi148:> grep smithi148.front.sepia.ceph.com /etc/hosts 2021-01-15T18:30:55.276 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2021-01-15T18:30:55.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2021-01-15T18:30:55.385 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hosts 2021-01-15T18:30:55.411 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/172.21.15.148/172.21.15.148/g' /etc/hosts 2021-01-15T18:30:55.476 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hostname 2021-01-15T18:30:55.540 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2021-01-15T18:30:55.604 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2021-01-15T18:30:55.605 DEBUG:teuthology.parallel:result is None 2021-01-15T18:30:55.702 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2021-01-15T18:30:55.702 INFO:teuthology.lock.ops:New key found. Updating... 2021-01-15T18:30:55.721 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2021-01-15T18:30:55.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T18:30:55.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:30:55.788 DEBUG:teuthology.orchestra.run.smithi148:> true 2021-01-15T18:30:55.944 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2021-01-15T18:30:55.997 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2021-01-15T18:30:55.998 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2021-01-15T18:30:56.051 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Red Hat Enterprise Linux" 2021-01-15T18:30:56.052 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8.3 (Ootpa)" 2021-01-15T18:30:56.052 INFO:teuthology.orchestra.run.smithi148.stdout:ID="rhel" 2021-01-15T18:30:56.052 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="fedora" 2021-01-15T18:30:56.053 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8.3" 2021-01-15T18:30:56.053 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2021-01-15T18:30:56.053 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-01-15T18:30:56.053 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2021-01-15T18:30:56.054 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-01-15T18:30:56.054 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.redhat.com/" 2021-01-15T18:30:56.054 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-01-15T18:30:56.054 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:30:56.054 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-01-15T18:30:56.055 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-01-15T18:30:56.055 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-01-15T18:30:56.055 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-01-15T18:30:56.056 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-01-15T18:30:56.071 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2021-01-15T18:30:56.088 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Ata3o+j5f9cAKtufgWCn2Fw+o3874hUg/H93OmKwtXSpLfnN/Nj7Eu4k2CYhV4xhQra9OSwoZkC8sJdfD8YPvnsVAi+EUfuPwSibz10lD3sbFpBNMjmqo5srhnP+WGvWSPXmkCU0Zmo/qgPI573m6K0U0JXaha/Wy739fsnMhwmcMtTffyQLvjbGJX20f45058oj/SHV2E4uqaaieURLvR7TCLk9FqbymTpXE3uhEKvXeTmkkBariAyHNuL8xV7sIzlypniKI4TXqEUGz2dJ6Ib32634ByNEt7fbenySQRxh0AwJJnmUFH0krB7cK5NH7X/JZeEWr7c4+NYFr54DMCcPol1nlrFSXQYWoA2J2CaT6vJpt2J9YMFB3FOm7cj2wlpo0QXkpHh+x/lhEjjxfosve6fdxkbhrC4oubCzIdWyE/xQgCIQ1Njo4WK6c7Qq7vkAP5Uz2X4m3IlfWfFks5FNFlmnoh8sX+61YuSh1Mj06y9sCXofFDa75IzrCF8= 2021-01-15T18:30:56.089 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T18:30:56.136 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-01-15T18:30:56.143 INFO:teuthology.task.internal:Saving configuration 2021-01-15T18:30:56.157 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-01-15T18:30:56.277 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-01-15T18:30:56.295 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-01-15 18:25:56.515027', 'description': '/home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI9aYiyzTcyaKWxDpYdI6cT8KU7mSNBzjndqTkhJzZztUh1WSSZYnWIq99oFUUYffvX+6WjYA7d4/mYWkbjW46M=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi148.front.sepia.ceph.com'} 2021-01-15T18:30:56.296 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-01-15T18:30:56.303 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-01-15T18:30:56.303 INFO:teuthology.run_tasks:Running task console_log... 2021-01-15T18:30:56.349 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60bb4efa60>, signals=[15]) 2021-01-15T18:30:56.349 INFO:teuthology.run_tasks:Running task internal.connect... 2021-01-15T18:30:56.446 INFO:teuthology.task.internal:Opening connections... 2021-01-15T18:30:56.447 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:30:56.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:30:56.520 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-01-15T18:30:56.564 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2021-01-15T18:30:56.678 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2021-01-15T18:30:56.678 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2021-01-15T18:30:56.731 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Red Hat Enterprise Linux" 2021-01-15T18:30:56.732 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8.3 (Ootpa)" 2021-01-15T18:30:56.732 INFO:teuthology.orchestra.run.smithi148.stdout:ID="rhel" 2021-01-15T18:30:56.732 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="fedora" 2021-01-15T18:30:56.733 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8.3" 2021-01-15T18:30:56.733 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2021-01-15T18:30:56.733 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-01-15T18:30:56.734 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2021-01-15T18:30:56.734 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-01-15T18:30:56.734 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.redhat.com/" 2021-01-15T18:30:56.735 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-01-15T18:30:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:30:56.735 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-01-15T18:30:56.735 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-01-15T18:30:56.736 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-01-15T18:30:56.736 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-01-15T18:30:56.736 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-01-15T18:30:56.752 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-01-15T18:30:56.794 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-01-15T18:30:56.805 INFO:teuthology.task.internal:Checking for old test directory... 2021-01-15T18:30:56.806 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2021-01-15T18:30:56.819 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-01-15T18:30:56.829 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-01-15T18:30:56.830 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2021-01-15T18:30:56.889 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-01-15T18:30:56.917 INFO:teuthology.run_tasks:Running task kernel... 2021-01-15T18:30:56.938 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}} 2021-01-15T18:30:56.939 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-01-15T18:30:56.939 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2021-01-15T18:30:56.955 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-01-15T18:30:56.955 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T18:30:56.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2021-01-15T18:30:58.429 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:30:58.432 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2021-01-15T18:30:58.556 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:30:58.556 INFO:teuthology.orchestra.run.smithi148.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-01-15T18:30:58.557 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:31:00.297 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 24 MB/s | 8.8 MB 00:00 2021-01-15T18:31:02.031 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 4.3 MB/s | 23 kB 00:00 2021-01-15T18:31:02.585 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-01-15T18:31:02.585 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-01-15T18:31:02.696 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:31:02.705 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:31:02.705 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:31:02.827 DEBUG:teuthology.orchestra.run.smithi148:> echo no | sudo yum reinstall kernel || true 2021-01-15T18:31:03.091 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:31:03.091 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2021-01-15T18:31:03.139 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:31:03.140 INFO:teuthology.orchestra.run.smithi148.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-01-15T18:31:03.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:31:03.307 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:01 ago on Fri 15 Jan 2021 06:31:02 PM UTC. 2021-01-15T18:31:03.459 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-01-15T18:31:03.460 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-01-15T18:31:03.502 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2021-01-15T18:31:03.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum reinstall -y kernel || true 2021-01-15T18:31:03.801 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:31:03.801 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2021-01-15T18:31:03.844 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:31:03.844 INFO:teuthology.orchestra.run.smithi148.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-01-15T18:31:03.845 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:31:04.012 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:02 ago on Fri 15 Jan 2021 06:31:02 PM UTC. 2021-01-15T18:31:04.183 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-01-15T18:31:04.184 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-01-15T18:31:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2021-01-15T18:31:04.251 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel --last | head -n 1 2021-01-15T18:31:04.339 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-01-15T18:31:04.340 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T18:31:04.340 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-01-15T18:31:04.341 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-01-15T18:31:04.341 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-01-15T18:31:04.375 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2021-01-15T18:31:04.390 INFO:teuthology.run_tasks:Running task internal.base... 2021-01-15T18:31:04.403 INFO:teuthology.task.internal:Creating test directory... 2021-01-15T18:31:04.403 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-01-15T18:31:04.443 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-01-15T18:31:04.452 INFO:teuthology.run_tasks:Running task internal.archive... 2021-01-15T18:31:04.465 INFO:teuthology.task.internal:Creating archive directory... 2021-01-15T18:31:04.466 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-01-15T18:31:04.510 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-01-15T18:31:04.520 INFO:teuthology.task.internal:Enabling coredump saving... 2021-01-15T18:31:04.522 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-01-15T18:31:04.580 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:31:04.590 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:31:04.591 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-01-15T18:31:04.600 INFO:teuthology.task.internal:Configuring sudo... 2021-01-15T18:31:04.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-01-15T18:31:04.655 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-01-15T18:31:04.668 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-01-15T18:31:04.668 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-01-15T18:31:04.709 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-01-15T18:31:04.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-01-15T18:31:04.834 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-01-15T18:31:04.908 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-01-15T18:31:04.953 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:31:04.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-01-15T18:31:05.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2021-01-15T18:31:05.124 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-01-15T18:31:05.467 INFO:teuthology.run_tasks:Running task internal.timer... 2021-01-15T18:31:05.478 INFO:teuthology.task.internal:Starting timer... 2021-01-15T18:31:05.479 INFO:teuthology.run_tasks:Running task pcp... 2021-01-15T18:31:05.493 INFO:teuthology.run_tasks:Running task selinux... 2021-01-15T18:31:05.506 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-01-15T18:31:05.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2021-01-15T18:31:05.587 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: [ OK ] 2021-01-15T18:31:05.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-01-15T18:31:05.589 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2021-01-15T18:31:05.633 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2021-01-15T18:31:05.634 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2021-01-15T18:31:05.634 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-01-15T18:31:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:31:05.659 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-01-15T18:31:05.659 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2021-01-15T18:31:05.723 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-01-15T18:31:05.740 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-01-15T18:31:05.746 INFO:teuthology.repo_utils:Fetching master from origin 2021-01-15T18:31:05.786 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2021-01-15T18:31:05.798 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-01-15T18:31:05.799 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-01-15T18:31:06.128 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-01-15T18:31:07.006 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-01-15T18:31:09.488 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-01-15T18:31:09.508 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:09 +0000 (0:00:00.069) 0:00:00.069 ******** 2021-01-15T18:31:10.914 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi148.front.sepia.ceph.com] 2021-01-15T18:31:10.983 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:10 +0000 (0:00:01.474) 0:00:01.544 ******** 2021-01-15T18:31:11.287 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-01-15T18:31:11.289 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-01-15T18:31:11.320 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:11 +0000 (0:00:00.336) 0:00:01.881 ******** 2021-01-15T18:31:11.728 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-01-15T18:31:11.730 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-01-15T18:31:11.761 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:11 +0000 (0:00:00.441) 0:00:02.323 ******** 2021-01-15T18:31:11.787 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-01-15T18:31:11.789 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:11.819 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:11 +0000 (0:00:00.058) 0:00:02.381 ******** 2021-01-15T18:31:12.116 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-01-15T18:31:12.118 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T18:31:12.151 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:12 +0000 (0:00:00.331) 0:00:02.712 ******** 2021-01-15T18:31:13.044 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-01-15T18:31:13.046 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610735472.1649415-185053171817303/source", "state": "file", "uid": 0} 2021-01-15T18:31:13.077 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:13 +0000 (0:00:00.926) 0:00:03.639 ******** 2021-01-15T18:31:13.538 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-01-15T18:31:13.540 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-01-15T18:31:13.541 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-01-15T18:31:13.627 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-01-15T18:31:13.634 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-01-15T18:31:13.782 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:13 +0000 (0:00:00.704) 0:00:04.343 ******** 2021-01-15T18:31:14.400 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi148.front.sepia.ceph.com] 2021-01-15T18:31:14.474 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-01-15T18:31:14.475 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:14 +0000 (0:00:00.691) 0:00:05.035 ******** 2021-01-15T18:31:14.969 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-01-15T18:31:14.971 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.178947", "end": "2021-01-15 18:31:14.947964", "rc": 0, "start": "2021-01-15 18:31:14.769017", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-01-15T18:31:15.005 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:14 +0000 (0:00:00.530) 0:00:05.566 ******** 2021-01-15T18:31:15.189 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-01-15T18:31:15.192 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009530", "end": "2021-01-15 18:31:15.173468", "failed_when_result": false, "rc": 0, "start": "2021-01-15 18:31:15.163938", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-01-15T18:31:15.224 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.219) 0:00:05.786 ******** 2021-01-15T18:31:15.251 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-01-15T18:31:15.253 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:15.285 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.060) 0:00:05.846 ******** 2021-01-15T18:31:15.311 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:15.358 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.072) 0:00:05.919 ******** 2021-01-15T18:31:15.384 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-01-15T18:31:15.386 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:15.419 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.061) 0:00:05.980 ******** 2021-01-15T18:31:15.445 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-01-15T18:31:15.447 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:15.481 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.062) 0:00:06.042 ******** 2021-01-15T18:31:15.507 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-01-15T18:31:15.509 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:15.540 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.058) 0:00:06.101 ******** 2021-01-15T18:31:15.732 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-01-15T18:31:15.733 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T18:31:15.764 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:15 +0000 (0:00:00.224) 0:00:06.325 ******** 2021-01-15T18:31:16.001 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-01-15T18:31:16.003 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-01-15T18:31:16.200 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-01-15T18:31:16.387 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-01-15T18:31:16.566 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-01-15T18:31:16.744 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-01-15T18:31:16.935 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-01-15T18:31:17.123 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-01-15T18:31:17.301 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-01-15T18:31:17.480 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-01-15T18:31:17.668 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-01-15T18:31:17.844 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-01-15T18:31:18.024 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-01-15T18:31:18.202 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-01-15T18:31:18.381 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-01-15T18:31:18.568 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-01-15T18:31:18.746 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-01-15T18:31:18.925 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-01-15T18:31:19.103 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-01-15T18:31:19.282 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-01-15T18:31:19.476 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-01-15T18:31:19.656 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-01-15T18:31:19.846 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-01-15T18:31:20.038 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-01-15T18:31:20.237 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-01-15T18:31:20.446 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-01-15T18:31:20.639 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-01-15T18:31:20.842 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-01-15T18:31:21.020 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-01-15T18:31:21.199 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-01-15T18:31:21.389 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-01-15T18:31:21.578 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-01-15T18:31:21.758 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-01-15T18:31:21.946 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-01-15T18:31:22.125 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-01-15T18:31:22.303 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-01-15T18:31:22.490 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-01-15T18:31:22.666 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-01-15T18:31:22.845 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-01-15T18:31:23.026 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-01-15T18:31:23.216 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-01-15T18:31:23.415 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-01-15T18:31:23.594 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-01-15T18:31:23.771 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-01-15T18:31:23.957 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-01-15T18:31:24.134 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-01-15T18:31:24.136 INFO:teuthology.task.ansible.out: 2021-01-15T18:31:24.322 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-01-15T18:31:24.498 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-01-15T18:31:24.684 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-01-15T18:31:24.861 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-01-15T18:31:25.039 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-01-15T18:31:25.219 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-01-15T18:31:25.397 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-01-15T18:31:25.620 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-01-15T18:31:25.828 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-01-15T18:31:26.014 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-01-15T18:31:26.201 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-01-15T18:31:26.380 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-01-15T18:31:26.557 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-01-15T18:31:26.733 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-01-15T18:31:26.919 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-01-15T18:31:27.098 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-01-15T18:31:27.275 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-01-15T18:31:27.451 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-01-15T18:31:27.639 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-01-15T18:31:27.817 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-01-15T18:31:27.995 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-01-15T18:31:28.172 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-01-15T18:31:28.350 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-01-15T18:31:28.526 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-01-15T18:31:28.716 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-01-15T18:31:28.892 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-01-15T18:31:29.070 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-01-15T18:31:29.248 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-01-15T18:31:29.430 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-01-15T18:31:29.618 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-01-15T18:31:29.805 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-01-15T18:31:29.983 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-01-15T18:31:30.172 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-01-15T18:31:30.351 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-01-15T18:31:30.526 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-01-15T18:31:30.713 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-01-15T18:31:30.890 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-01-15T18:31:31.066 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-01-15T18:31:31.255 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-01-15T18:31:31.431 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-01-15T18:31:31.618 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-01-15T18:31:31.805 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-01-15T18:31:31.982 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-01-15T18:31:32.169 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-01-15T18:31:32.345 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-01-15T18:31:32.521 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-01-15T18:31:32.707 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-01-15T18:31:32.883 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-01-15T18:31:33.074 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-01-15T18:31:33.250 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-01-15T18:31:33.427 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-01-15T18:31:33.603 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-01-15T18:31:33.790 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-01-15T18:31:33.966 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-01-15T18:31:34.143 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-01-15T18:31:34.330 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-01-15T18:31:34.506 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-01-15T18:31:34.682 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-01-15T18:31:34.868 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-01-15T18:31:35.044 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-01-15T18:31:35.223 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-01-15T18:31:35.410 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-01-15T18:31:35.586 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-01-15T18:31:35.773 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-01-15T18:31:35.949 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-01-15T18:31:36.126 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-01-15T18:31:36.312 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-01-15T18:31:36.499 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-01-15T18:31:36.675 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-01-15T18:31:36.851 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-01-15T18:31:37.027 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-01-15T18:31:37.216 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-01-15T18:31:37.393 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-01-15T18:31:37.570 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-01-15T18:31:37.758 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-01-15T18:31:37.933 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-01-15T18:31:38.119 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-01-15T18:31:38.306 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-01-15T18:31:38.493 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-01-15T18:31:38.668 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-01-15T18:31:38.844 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-01-15T18:31:39.030 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-01-15T18:31:39.210 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-01-15T18:31:39.387 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-01-15T18:31:39.556 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-01-15T18:31:39.744 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-01-15T18:31:39.921 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-01-15T18:31:40.099 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-01-15T18:31:40.286 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-01-15T18:31:40.464 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-01-15T18:31:40.642 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-01-15T18:31:42.156 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-01-15T18:31:43.934 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-01-15T18:31:45.523 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-01-15T18:31:45.626 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:45 +0000 (0:00:29.861) 0:00:36.187 ******** 2021-01-15T18:31:45.651 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-01-15T18:31:45.684 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:45 +0000 (0:00:00.058) 0:00:36.245 ******** 2021-01-15T18:31:45.722 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-01-15T18:31:45.724 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-01-15T18:31:45.725 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-01-15T18:31:45.727 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-01-15T18:31:45.729 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkK 2021-01-15T18:31:45.730 INFO:teuthology.task.ansible.out:ztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop 2021-01-15T18:31:45.731 INFO:teuthology.task.ansible.out: XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2021-01-15T18:31:45.732 INFO:teuthology.task.ansible.out:HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2021-01-15T18:31:45.734 INFO:teuthology.task.ansible.out:e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df 2021-01-15T18:31:45.736 INFO:teuthology.task.ansible.out:79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hm 2021-01-15T18:31:45.737 INFO:teuthology.task.ansible.out:p+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b 2021-01-15T18:31:45.739 INFO:teuthology.task.ansible.out:0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "z 2021-01-15T18:31:45.739 INFO:teuthology.task.ansible.out:dover@lenovot5301 88ArNhsAM 2021-01-15T18:31:45.741 INFO:teuthology.task.ansible.out:Xq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jel 2021-01-15T18:31:45.742 INFO:teuthology.task.ansible.out:opez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf 2021-01-15T18:31:45.744 INFO:teuthology.task.ansible.out:0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}]}, "changed": false} 2021-01-15T18:31:45.780 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:45 +0000 (0:00:00.095) 0:00:36.341 ******** 2021-01-15T18:31:46.595 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-01-15T18:31:46.597 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com -> localhost] => {"after": "0caff10a478f2674721da1965013b225f3ea5800", "attempts": 1, "before": "0caff10a478f2674721da1965013b225f3ea5800", "changed": false, "remote_url_changed": false} 2021-01-15T18:31:46.629 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:31:46 +0000 (0:00:00.849) 0:00:37.191 ******** 2021-01-15T18:31:46.924 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-01-15T18:31:46.926 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-01-15T18:31:47.165 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-01-15T18:31:47.402 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-01-15T18:31:47.631 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-01-15T18:31:47.633 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-01-15T18:31:47.870 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-01-15T18:31:48.099 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-01-15T18:31:48.100 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-01-15T18:31:48.347 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-01-15T18:31:48.607 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-01-15T18:31:48.836 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-01-15T18:31:48.847 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:49.076 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-01-15T18:31:49.078 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-01-15T18:31:49.305 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-01-15T18:31:49.307 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:49.317 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:31:49.568 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-01-15T18:31:49.786 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-01-15T18:31:49.788 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-01-15T18:31:50.014 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-01-15T18:31:50.232 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-01-15T18:31:50.452 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-01-15T18:31:50.682 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-01-15T18:31:50.911 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-01-15T18:31:51.131 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2021-01-15T18:31:51.369 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-01-15T18:31:51.371 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-01-15T18:31:51.613 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-01-15T18:31:51.615 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-01-15T18:31:51.850 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-01-15T18:31:52.095 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-01-15T18:31:52.325 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-01-15T18:31:52.327 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-01-15T18:31:52.546 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-01-15T18:31:52.780 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-01-15T18:31:53.009 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-01-15T18:31:53.239 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-01-15T18:31:53.473 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-01-15T18:31:53.692 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-01-15T18:31:53.694 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-01-15T18:31:53.911 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-01-15T18:31:54.150 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-01-15T18:31:54.380 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-01-15T18:31:54.609 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-01-15T18:31:54.828 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-01-15T18:31:55.047 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-01-15T18:31:55.266 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-01-15T18:31:55.485 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-01-15T18:31:55.725 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-01-15T18:31:55.946 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-01-15T18:31:55.947 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-01-15T18:31:56.176 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-01-15T18:31:56.177 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-01-15T18:31:56.395 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-01-15T18:31:56.624 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-01-15T18:31:56.853 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-01-15T18:31:57.082 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-01-15T18:31:57.084 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-01-15T18:31:57.310 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-01-15T18:31:57.312 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-01-15T18:31:57.314 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-01-15T18:31:57.554 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-01-15T18:31:57.784 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-01-15T18:31:58.015 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-01-15T18:31:58.235 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-01-15T18:31:58.464 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-01-15T18:31:58.694 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-01-15T18:31:58.695 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-01-15T18:31:58.913 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-01-15T18:31:59.143 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-01-15T18:31:59.363 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-01-15T18:31:59.365 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-01-15T18:31:59.594 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-01-15T18:31:59.825 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-01-15T18:32:00.044 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-01-15T18:32:00.046 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-01-15T18:32:00.275 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-01-15T18:32:00.504 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-01-15T18:32:00.722 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-01-15T18:32:00.940 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-01-15T18:32:01.167 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-01-15T18:32:01.398 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-01-15T18:32:01.619 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-01-15T18:32:01.849 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-01-15T18:32:02.080 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-01-15T18:32:02.301 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-01-15T18:32:02.524 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-01-15T18:32:02.752 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-01-15T18:32:02.972 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-01-15T18:32:03.192 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-01-15T18:32:03.421 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-01-15T18:32:03.653 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-01-15T18:32:03.894 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-01-15T18:32:04.126 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-01-15T18:32:04.358 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-01-15T18:32:04.588 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-01-15T18:32:04.818 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-01-15T18:32:04.820 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-01-15T18:32:05.039 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-01-15T18:32:05.286 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-01-15T18:32:05.522 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-01-15T18:32:05.765 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-01-15T18:32:05.992 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-01-15T18:32:05.994 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-01-15T18:32:06.223 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-01-15T18:32:06.224 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-01-15T18:32:06.443 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-01-15T18:32:06.683 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-01-15T18:32:06.921 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-01-15T18:32:07.143 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-01-15T18:32:07.363 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-01-15T18:32:07.592 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-01-15T18:32:07.821 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-01-15T18:32:08.050 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-01-15T18:32:08.052 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-01-15T18:32:08.282 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-01-15T18:32:08.540 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-01-15T18:32:08.795 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-01-15T18:32:09.058 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-01-15T18:32:09.312 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-01-15T18:32:09.549 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-01-15T18:32:09.789 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-01-15T18:32:09.791 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-01-15T18:32:10.021 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-01-15T18:32:10.251 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-01-15T18:32:10.471 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-01-15T18:32:10.699 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-01-15T18:32:10.918 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-01-15T18:32:11.146 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-01-15T18:32:11.378 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-01-15T18:32:11.606 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-01-15T18:32:11.834 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-01-15T18:32:12.062 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-01-15T18:32:12.280 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-01-15T18:32:12.509 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-01-15T18:32:12.739 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-01-15T18:32:12.957 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-01-15T18:32:13.187 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-01-15T18:32:13.419 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-01-15T18:32:13.649 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-01-15T18:32:13.859 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-01-15T18:32:14.088 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-01-15T18:32:14.090 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-01-15T18:32:14.316 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-01-15T18:32:14.545 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-01-15T18:32:14.765 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-01-15T18:32:14.994 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-01-15T18:32:15.222 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-01-15T18:32:15.444 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-01-15T18:32:15.673 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-01-15T18:32:15.944 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-01-15T18:32:16.164 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-01-15T18:32:16.404 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-01-15T18:32:16.623 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-01-15T18:32:16.853 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-01-15T18:32:17.072 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-01-15T18:32:17.290 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-01-15T18:32:17.520 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-01-15T18:32:17.737 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-01-15T18:32:17.976 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-01-15T18:32:18.074 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:18 +0000 (0:00:31.444) 0:01:08.635 ******** 2021-01-15T18:32:18.111 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.122 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.124 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.134 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.145 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.147 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.156 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.158 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.168 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.409 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-01-15T18:32:18.410 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.420 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.642 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-01-15T18:32:18.863 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-01-15T18:32:18.874 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.885 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.905 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.916 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.918 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.927 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.938 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.940 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.950 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.961 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.975 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.985 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.986 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:18.997 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.008 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.010 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.019 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.030 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.041 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.043 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.052 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.063 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.065 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.075 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.086 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.088 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.098 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.109 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.110 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.120 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.132 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.133 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.144 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.155 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.157 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.167 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.168 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.178 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.189 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.190 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.202 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.213 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.213 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.225 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.236 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.237 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.247 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.258 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.259 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.269 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.280 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.282 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.291 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.303 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.304 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.315 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.315 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.326 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.337 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.339 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.349 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.360 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.371 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.382 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.383 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.393 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.404 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.415 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.417 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.426 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.428 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.438 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.449 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.451 INFO:teuthology.task.ansible.out: skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.462 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.475 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.476 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.486 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.496 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.498 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.507 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.518 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.520 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.529 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.543 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.551 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.562 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.563 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.573 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.584 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.585 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.595 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.606 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.607 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.617 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.628 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.639 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.640 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.650 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.660 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.662 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.672 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.683 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.694 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.705 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.715 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.726 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.728 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.737 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.748 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.750 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.759 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.770 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.772 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.781 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.792 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.794 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.803 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.814 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.825 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.826 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.874 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:19 +0000 (0:00:01.799) 0:01:10.435 ******** 2021-01-15T18:32:19.899 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:19.931 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:19 +0000 (0:00:00.057) 0:01:10.492 ******** 2021-01-15T18:32:20.145 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-01-15T18:32:20.147 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-01-15T18:32:20.324 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-01-15T18:32:20.503 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-01-15T18:32:20.679 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-01-15T18:32:20.846 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-01-15T18:32:21.023 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-01-15T18:32:21.211 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-01-15T18:32:21.387 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-01-15T18:32:21.555 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-01-15T18:32:21.731 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-01-15T18:32:21.908 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-01-15T18:32:22.086 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-01-15T18:32:22.263 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-01-15T18:32:22.441 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-01-15T18:32:22.618 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-01-15T18:32:22.796 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-01-15T18:32:22.973 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-01-15T18:32:23.141 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-01-15T18:32:23.318 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-01-15T18:32:23.498 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-01-15T18:32:23.685 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-01-15T18:32:23.861 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-01-15T18:32:24.037 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-01-15T18:32:24.225 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-01-15T18:32:24.403 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-01-15T18:32:24.578 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-01-15T18:32:24.754 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-01-15T18:32:24.929 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-01-15T18:32:25.105 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-01-15T18:32:25.281 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-01-15T18:32:25.462 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-01-15T18:32:25.637 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-01-15T18:32:25.804 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-01-15T18:32:25.992 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-01-15T18:32:26.169 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-01-15T18:32:26.334 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-01-15T18:32:26.511 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-01-15T18:32:28.049 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-01-15T18:32:28.238 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-01-15T18:32:28.417 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-01-15T18:32:28.598 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-01-15T18:32:28.776 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-01-15T18:32:28.952 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-01-15T18:32:29.129 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-01-15T18:32:29.307 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-01-15T18:32:29.485 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-01-15T18:32:29.665 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-01-15T18:32:29.843 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-01-15T18:32:29.906 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:29 +0000 (0:00:09.974) 0:01:20.467 ******** 2021-01-15T18:32:30.409 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-01-15T18:32:30.411 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "checksum": "e880c991f1562524752e643d227b8575ac6e2ed1", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "fd5444fc24d885aa1caaed5c7ef528f8", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610735549.9280066-107761748624634/source", "state": "file", "uid": 0} 2021-01-15T18:32:30.443 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:30 +0000 (0:00:00.537) 0:01:21.004 ******** 2021-01-15T18:32:30.469 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-01-15T18:32:30.471 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => { "msg": "Host smithi148.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-01-15T18:32:30.503 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:30 +0000 (0:00:00.059) 0:01:21.064 ******** 2021-01-15T18:32:30.532 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-01-15T18:32:30.534 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-01-15T18:32:30.563 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:30 +0000 (0:00:00.060) 0:01:21.125 ******** 2021-01-15T18:32:30.589 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-01-15T18:32:30.719 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 15 January 2021 18:32:30 +0000 (0:00:00.057) 0:01:21.183 ******** 2021-01-15T18:32:31.130 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-01-15T18:32:31.132 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.320372", "end": "2021-01-15 18:32:31.111916", "rc": 0, "start": "2021-01-15 18:32:30.791544", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-01-15T18:32:31.163 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.541) 0:01:21.724 ******** 2021-01-15T18:32:31.189 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-01-15T18:32:31.191 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:31.223 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.059) 0:01:21.784 ******** 2021-01-15T18:32:31.249 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:31.281 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.058) 0:01:21.842 ******** 2021-01-15T18:32:31.307 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-01-15T18:32:31.308 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:31.354 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.073) 0:01:21.916 ******** 2021-01-15T18:32:31.382 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-01-15T18:32:31.384 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:31.414 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.059) 0:01:21.975 ******** 2021-01-15T18:32:31.440 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-01-15T18:32:31.442 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:31.472 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.057) 0:01:22.033 ******** 2021-01-15T18:32:31.508 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-01-15T18:32:31.509 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-01-15T18:32:31.540 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.068) 0:01:22.102 ******** 2021-01-15T18:32:31.587 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-01-15T18:32:31.589 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T18:32:31.620 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.080) 0:01:22.182 ******** 2021-01-15T18:32:31.668 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-01-15T18:32:31.670 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-01-15T18:32:31.705 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:31 +0000 (0:00:00.084) 0:01:22.266 ******** 2021-01-15T18:32:32.894 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-01-15T18:32:32.896 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1610735552.766845-eyf2yed0/katello-ca-consumer-latest.noarchxj8gydm9.rpm"]} 2021-01-15T18:32:32.929 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:32 +0000 (0:00:01.224) 0:01:23.490 ******** 2021-01-15T18:32:33.481 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-01-15T18:32:33.516 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:33 +0000 (0:00:00.586) 0:01:24.077 ******** 2021-01-15T18:32:33.554 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-01-15T18:32:33.556 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-01-15T18:32:33.587 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:33 +0000 (0:00:00.071) 0:01:24.149 ******** 2021-01-15T18:32:49.440 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-01-15T18:32:49.442 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-01-15T18:32:49.473 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:49 +0000 (0:00:15.885) 0:01:40.034 ******** 2021-01-15T18:32:49.510 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-01-15T18:32:49.511 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-01-15T18:32:49.543 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:49 +0000 (0:00:00.070) 0:01:40.105 ******** 2021-01-15T18:32:50.099 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-01-15T18:32:50.101 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.364215", "end": "2021-01-15 18:32:50.070955", "failed_when_result": false, "rc": 0, "start": "2021-01-15 18:32:49.706740", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-01-15T18:32:50.134 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:50 +0000 (0:00:00.590) 0:01:40.696 ******** 2021-01-15T18:32:50.160 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-01-15T18:32:50.162 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:50.194 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:50 +0000 (0:00:00.059) 0:01:40.756 ******** 2021-01-15T18:32:50.222 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-01-15T18:32:50.224 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:50.256 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:50 +0000 (0:00:00.062) 0:01:40.818 ******** 2021-01-15T18:32:50.283 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-01-15T18:32:50.284 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T18:32:50.315 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:50 +0000 (0:00:00.059) 0:01:40.877 ******** 2021-01-15T18:32:50.352 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-01-15T18:32:50.354 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:50.385 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:50 +0000 (0:00:00.069) 0:01:40.946 ******** 2021-01-15T18:32:52.942 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-01-15T18:32:52.944 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.374386", "end": "2021-01-15 18:32:52.924255", "rc": 0, "start": "2021-01-15 18:32:50.549869", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-01-15T18:32:52.981 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:52 +0000 (0:00:02.596) 0:01:43.542 ******** 2021-01-15T18:32:53.028 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-01-15T18:32:53.030 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-01-15T18:32:53.063 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:53 +0000 (0:00:00.082) 0:01:43.624 ******** 2021-01-15T18:32:53.089 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-01-15T18:32:53.091 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:32:53.124 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:53 +0000 (0:00:00.060) 0:01:43.685 ******** 2021-01-15T18:32:53.160 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-01-15T18:32:53.162 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-01-15T18:32:53.199 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:53 +0000 (0:00:00.075) 0:01:43.760 ******** 2021-01-15T18:32:53.247 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-01-15T18:32:53.283 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:53 +0000 (0:00:00.083) 0:01:43.844 ******** 2021-01-15T18:32:55.308 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-01-15T18:32:55.309 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-01-15T18:32:55.344 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:55 +0000 (0:00:02.061) 0:01:45.906 ******** 2021-01-15T18:32:58.449 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-01-15T18:32:58.451 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:02.915652", "end": "2021-01-15 18:32:58.423635", "rc": 0, "start": "2021-01-15 18:32:55.507983", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-01-15T18:32:58.485 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:58 +0000 (0:00:03.140) 0:01:49.046 ******** 2021-01-15T18:32:58.917 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-01-15T18:32:58.919 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-01-15T18:32:58.950 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:58 +0000 (0:00:00.465) 0:01:49.512 ******** 2021-01-15T18:32:59.131 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-01-15T18:32:59.134 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T18:32:59.163 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:32:59 +0000 (0:00:00.212) 0:01:49.724 ******** 2021-01-15T18:32:59.615 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-01-15T18:32:59.617 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-01-15T18:33:00.031 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-01-15T18:33:00.066 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:00 +0000 (0:00:00.902) 0:01:50.627 ******** 2021-01-15T18:33:00.510 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-01-15T18:33:00.512 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-01-15T18:33:00.934 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-01-15T18:33:00.969 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:00 +0000 (0:00:00.903) 0:01:51.531 ******** 2021-01-15T18:33:01.006 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-01-15T18:33:01.008 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.051 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.081) 0:01:51.612 ******** 2021-01-15T18:33:01.078 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-01-15T18:33:01.079 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.110 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.059) 0:01:51.672 ******** 2021-01-15T18:33:01.136 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-01-15T18:33:01.138 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.171 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.060) 0:01:51.732 ******** 2021-01-15T18:33:01.197 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-01-15T18:33:01.198 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.230 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.059) 0:01:51.792 ******** 2021-01-15T18:33:01.257 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-01-15T18:33:01.259 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.291 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.060) 0:01:51.853 ******** 2021-01-15T18:33:01.317 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-01-15T18:33:01.319 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.353 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.062) 0:01:51.915 ******** 2021-01-15T18:33:01.380 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-01-15T18:33:01.382 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.415 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.061) 0:01:51.976 ******** 2021-01-15T18:33:01.441 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-01-15T18:33:01.442 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.475 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.060) 0:01:52.036 ******** 2021-01-15T18:33:01.504 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-01-15T18:33:01.506 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:01.537 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:01 +0000 (0:00:00.061) 0:01:52.098 ******** 2021-01-15T18:33:02.551 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-01-15T18:33:02.553 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.832415", "end": "2021-01-15 18:33:02.527232", "rc": 0, "start": "2021-01-15 18:33:01.694817", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:33:02.586 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:02 +0000 (0:00:01.048) 0:01:53.147 ******** 2021-01-15T18:33:05.465 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-01-15T18:33:05.466 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2021-01-15T18:33:05.497 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:05 +0000 (0:00:02.910) 0:01:56.058 ******** 2021-01-15T18:33:05.524 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:05.567 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:05 +0000 (0:00:00.070) 0:01:56.128 ******** 2021-01-15T18:33:05.594 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-01-15T18:33:05.596 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:05.645 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:05 +0000 (0:00:00.077) 0:01:56.206 ******** 2021-01-15T18:33:06.019 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 15 January 2021 18:33:05 +0000 (0:00:00.066) 0:01:56.272 ******** TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 15 January 2021 18:33:05 +0000 (0:00:00.080) 0:01:56.353 ******** 2021-01-15T18:33:06.255 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-01-15T18:33:06.257 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-01-15T18:33:06.288 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:06 +0000 (0:00:00.496) 0:01:56.850 ******** 2021-01-15T18:33:08.778 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-01-15T18:33:08.780 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2021-01-15T18:33:08.812 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:08 +0000 (0:00:02.523) 0:01:59.373 ******** 2021-01-15T18:33:09.275 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-01-15T18:33:09.277 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-01-15T18:33:09.693 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-01-15T18:33:10.099 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-01-15T18:33:10.138 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:10 +0000 (0:00:01.326) 0:02:00.699 ******** 2021-01-15T18:33:10.321 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-01-15T18:33:10.323 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T18:33:10.355 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:10 +0000 (0:00:00.216) 0:02:00.916 ******** 2021-01-15T18:33:10.888 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-01-15T18:33:10.890 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610735590.3815567-127906098404620/source", "state": "file", "uid": 0} 2021-01-15T18:33:11.385 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-01-15T18:33:11.822 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-01-15T18:33:11.859 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:11 +0000 (0:00:01.504) 0:02:02.420 ******** 2021-01-15T18:33:11.891 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-01-15T18:33:11.894 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:11.925 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:11 +0000 (0:00:00.065) 0:02:02.486 ******** 2021-01-15T18:33:11.962 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-01-15T18:33:11.995 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:11 +0000 (0:00:00.070) 0:02:02.556 ******** 2021-01-15T18:33:12.199 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-01-15T18:33:12.231 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:12 +0000 (0:00:00.236) 0:02:02.793 ******** 2021-01-15T18:33:12.278 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-01-15T18:33:12.280 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:12.289 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:12.310 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:12.321 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:12.358 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:12 +0000 (0:00:00.126) 0:02:02.919 ******** 2021-01-15T18:33:14.934 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2021-01-15T18:33:14.968 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:14 +0000 (0:00:02.610) 0:02:05.529 ******** 2021-01-15T18:33:14.996 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:15.031 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:15 +0000 (0:00:00.063) 0:02:05.592 ******** 2021-01-15T18:33:15.466 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-01-15T18:33:15.468 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-01-15T18:33:15.501 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:15 +0000 (0:00:00.470) 0:02:06.062 ******** 2021-01-15T18:33:15.937 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-01-15T18:33:15.938 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-01-15T18:33:15.970 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:15 +0000 (0:00:00.469) 0:02:06.532 ******** 2021-01-15T18:33:16.008 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-01-15T18:33:16.011 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:16.020 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-01-15T18:33:16.053 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:16 +0000 (0:00:00.082) 0:02:06.614 ******** 2021-01-15T18:33:16.258 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-01-15T18:33:16.260 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T18:33:16.291 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:16 +0000 (0:00:00.237) 0:02:06.852 ******** 2021-01-15T18:33:16.571 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-01-15T18:33:16.572 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-01-15T18:33:16.605 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:16 +0000 (0:00:00.313) 0:02:07.166 ******** 2021-01-15T18:33:16.642 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-01-15T18:33:16.644 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:16.675 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:16 +0000 (0:00:00.070) 0:02:07.236 ******** 2021-01-15T18:33:17.119 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-01-15T18:33:17.121 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-01-15T18:33:17.153 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:17 +0000 (0:00:00.477) 0:02:07.714 ******** 2021-01-15T18:33:17.838 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-01-15T18:33:17.839 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-01-15 18:30:23 UTC", "ActiveEnterTimestampMonotonic": "15774527", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket system.slice local-fs.target -.mount network.target sysinit.target tmp.mount nss-lookup.target time-sync.target basic.target var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-01-15 18:30:23 UTC", "AssertTimestampMonotonic": "15773670", "Before": "xdm.service plymouth-quit.service getty@tty1.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-01-15 18:30:23 UTC", "ConditionTimestampMonotonic": "15773670", "ConfigurationDirectoryMode": " 2021-01-15T18:33:17.840 INFO:teuthology.task.ansible.out:0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "995", "ExecMainStartTimestamp": "Fri 2021-01-15 18:30:23 UTC", "ExecMainStartTimestampMonotonic": "15774495", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Fri 2021-01-15 18:30:23 UTC] ; stop_time=[n/a] ; pid=995 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-01-15 18:30:23 UTC", "InactiveExitTimestampMonotonic": "15774527", "InvocationID": "5d5445ea243e4a2594b25e9311f90cf5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2021-01-15T18:33:17.842 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "995", "MemoryAccounting": "yes", "MemoryCurrent": "1499136", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2021-01-15T18:33:17.843 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-01-15 18:30:23 UTC", "StateChangeTimestampMonotonic": "15774527", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-01-15 18:30:23 UTC", "WatchdogTimestampMonotonic": "15774526", "WatchdogUSec": "0"}} 2021-01-15T18:33:17.876 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:17 +0000 (0:00:00.722) 0:02:08.437 ******** 2021-01-15T18:33:18.064 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-01-15T18:33:18.066 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001705", "end": "2021-01-15 18:33:18.041164", "rc": 0, "start": "2021-01-15 18:33:18.039459", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-01-15T18:33:18.099 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:18 +0000 (0:00:00.223) 0:02:08.660 ******** 2021-01-15T18:33:20.635 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-01-15T18:33:20.636 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2021-01-15T18:33:20.668 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:20 +0000 (0:00:02.569) 0:02:11.230 ******** 2021-01-15T18:33:23.214 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-01-15T18:33:23.216 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2021-01-15T18:33:23.249 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:23 +0000 (0:00:02.580) 0:02:13.810 ******** 2021-01-15T18:33:24.132 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-01-15T18:33:24.133 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-01-15T18:33:24.167 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:24 +0000 (0:00:00.918) 0:02:14.728 ******** 2021-01-15T18:33:40.541 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-01-15T18:33:40.543 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.195141", "end": "2021-01-15 18:33:40.519839", "failed_when_result": false, "rc": 0, "start": "2021-01-15 18:33:24.324698", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-01-15T18:33:40.575 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:40 +0000 (0:00:16.408) 0:02:31.136 ******** 2021-01-15T18:33:41.019 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-01-15T18:33:41.021 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-01-15T18:33:41.053 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:41 +0000 (0:00:00.478) 0:02:31.615 ******** 2021-01-15T18:33:41.247 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-01-15T18:33:41.250 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010704", "end": "2021-01-15 18:33:41.225545", "rc": 0, "start": "2021-01-15 18:33:41.214841", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:33:41.281 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:41 +0000 (0:00:00.227) 0:02:31.842 ******** 2021-01-15T18:33:41.464 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-01-15T18:33:41.466 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002192", "end": "2021-01-15 18:33:41.443203", "rc": 0, "start": "2021-01-15 18:33:41.441011", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:33:41.499 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:41 +0000 (0:00:00.218) 0:02:32.061 ******** 2021-01-15T18:33:57.812 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-01-15T18:33:57.814 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.121786", "end": "2021-01-15 18:33:57.785196", "rc": 0, "start": "2021-01-15 18:33:41.663410", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:33:57.846 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:57 +0000 (0:00:16.346) 0:02:48.408 ******** 2021-01-15T18:33:58.031 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-01-15T18:33:58.033 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-01-15T18:33:58.075 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.228) 0:02:48.636 ******** 2021-01-15T18:33:58.104 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-01-15T18:33:58.106 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-01-15T18:33:58.153 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.077) 0:02:48.714 ******** 2021-01-15T18:33:58.201 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-01-15T18:33:58.203 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-01-15T18:33:58.238 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.084) 0:02:48.799 ******** 2021-01-15T18:33:58.274 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-01-15T18:33:58.276 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-01-15T18:33:58.310 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.072) 0:02:48.871 ******** 2021-01-15T18:33:58.350 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-01-15T18:33:58.352 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-01-15T18:33:58.383 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.072) 0:02:48.944 ******** 2021-01-15T18:33:58.566 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-01-15T18:33:58.568 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-01-15T18:33:58.599 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.216) 0:02:49.161 ******** 2021-01-15T18:33:58.781 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-01-15T18:33:58.783 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-01-15T18:33:58.814 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.214) 0:02:49.375 ******** 2021-01-15T18:33:58.840 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-01-15T18:33:58.842 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:58.875 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.060) 0:02:49.436 ******** 2021-01-15T18:33:58.902 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-01-15T18:33:58.904 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:33:58.954 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:58 +0000 (0:00:00.078) 0:02:49.515 ******** 2021-01-15T18:33:59.157 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-01-15T18:33:59.159 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-01-15T18:33:59.205 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:59 +0000 (0:00:00.251) 0:02:49.766 ******** 2021-01-15T18:33:59.721 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-01-15T18:33:59.722 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-01-15T18:33:59.753 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:59 +0000 (0:00:00.548) 0:02:50.315 ******** 2021-01-15T18:33:59.800 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-01-15T18:33:59.802 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi148"}, "changed": false} 2021-01-15T18:33:59.835 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:33:59 +0000 (0:00:00.081) 0:02:50.396 ******** 2021-01-15T18:34:00.813 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-01-15T18:34:00.815 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi148", "ansible_hostname": "smithi148", "ansible_nodename": "smithi148"}, "changed": false, "name": "smithi148"} 2021-01-15T18:34:00.845 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:00 +0000 (0:00:01.010) 0:02:51.406 ******** 2021-01-15T18:34:01.263 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-01-15T18:34:01.265 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-01-15T18:34:01.306 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:01 +0000 (0:00:00.461) 0:02:51.868 ******** 2021-01-15T18:34:01.738 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-01-15T18:34:01.740 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-01-15T18:34:01.771 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:01 +0000 (0:00:00.464) 0:02:52.332 ******** 2021-01-15T18:34:02.386 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-01-15T18:34:02.388 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-01-15T18:34:02.419 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:02 +0000 (0:00:00.648) 0:02:52.981 ******** 2021-01-15T18:34:02.857 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-01-15T18:34:02.858 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-01-15T18:34:02.890 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:02 +0000 (0:00:00.470) 0:02:53.451 ******** 2021-01-15T18:34:03.071 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-01-15T18:34:03.073 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-01-15T18:34:03.105 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:03 +0000 (0:00:00.215) 0:02:53.667 ******** 2021-01-15T18:34:03.286 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-01-15T18:34:03.288 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-01-15T18:34:03.320 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:03 +0000 (0:00:00.214) 0:02:53.881 ******** 2021-01-15T18:34:03.521 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-01-15T18:34:03.523 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-01-15T18:34:03.554 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:03 +0000 (0:00:00.234) 0:02:54.115 ******** 2021-01-15T18:34:03.981 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-01-15T18:34:03.983 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-01-15T18:34:04.013 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:04 +0000 (0:00:00.459) 0:02:54.575 ******** 2021-01-15T18:34:04.453 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-01-15T18:34:04.455 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-01-15T18:34:04.488 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:04 +0000 (0:00:00.474) 0:02:55.049 ******** 2021-01-15T18:34:06.650 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-01-15T18:34:06.652 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.979622", "end": "2021-01-15 18:34:06.625173", "rc": 0, "start": "2021-01-15 18:34:04.645551", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:06.682 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:06 +0000 (0:00:02.194) 0:02:57.243 ******** 2021-01-15T18:34:06.718 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-01-15T18:34:06.720 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:06.749 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:06 +0000 (0:00:00.067) 0:02:57.310 ******** 2021-01-15T18:34:07.208 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi148.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-01-15T18:34:07.243 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.494) 0:02:57.804 ******** 2021-01-15T18:34:07.269 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-01-15T18:34:07.303 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.060) 0:02:57.865 ******** 2021-01-15T18:34:07.330 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-01-15T18:34:07.362 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.059) 0:02:57.924 ******** 2021-01-15T18:34:07.388 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-01-15T18:34:07.390 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.421 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.058) 0:02:57.982 ******** 2021-01-15T18:34:07.458 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-01-15T18:34:07.460 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.469 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.501 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.080) 0:02:58.063 ******** 2021-01-15T18:34:07.538 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-01-15T18:34:07.540 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.572 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.070) 0:02:58.133 ******** 2021-01-15T18:34:07.599 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:34:07.600 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.631 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.059) 0:02:58.193 ******** 2021-01-15T18:34:07.668 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-01-15T18:34:07.670 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.704 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.072) 0:02:58.265 ******** 2021-01-15T18:34:07.741 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-01-15T18:34:07.742 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:07.774 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:07 +0000 (0:00:00.069) 0:02:58.335 ******** 2021-01-15T18:34:07.965 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-01-15T18:34:07.967 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006956", "end": "2021-01-15 18:34:07.943867", "msg": "non-zero return code", "rc": 1, "start": "2021-01-15 18:34:07.936911", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-01-15T18:34:08.003 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:08 +0000 (0:00:00.229) 0:02:58.564 ******** 2021-01-15T18:34:08.029 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-01-15T18:34:08.031 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:08.064 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:08 +0000 (0:00:00.061) 0:02:58.625 ******** 2021-01-15T18:34:14.038 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-01-15T18:34:14.040 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-01-15T18:34:14.071 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:14 +0000 (0:00:06.007) 0:03:04.633 ******** 2021-01-15T18:34:19.524 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-01-15T18:34:19.526 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-01-15T18:34:19.561 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:19 +0000 (0:00:05.489) 0:03:10.122 ******** 2021-01-15T18:34:31.353 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-01-15T18:34:31.355 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: make", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: lvm2", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: openssl-libs-1:1.1.1g-12.el8_3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: openssl-1:1.1.1g-12.el8_3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: openssl-1:1.1.1g-11.el8.x86_64", "Removed: openssl-libs-1:1.1.1g-11.el8.x86_64"]} 2021-01-15T18:34:31.387 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:31 +0000 (0:00:11.826) 0:03:21.948 ******** 2021-01-15T18:34:33.895 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-01-15T18:34:33.897 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: dbench"]} 2021-01-15T18:34:33.929 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:33 +0000 (0:00:02.541) 0:03:24.490 ******** 2021-01-15T18:34:36.508 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-01-15T18:34:36.510 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-01-15T18:34:36.541 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:36 +0000 (0:00:02.612) 0:03:27.102 ******** 2021-01-15T18:34:36.568 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-01-15T18:34:36.569 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:36.601 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:36 +0000 (0:00:00.060) 0:03:27.163 ******** 2021-01-15T18:34:36.890 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-01-15T18:34:36.891 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.service basic.target polkit.service sysinit.target dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nftables.service shutdown.target ebtables.service iptables.service ip6tables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-01-15T18:34:36.893 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-01-15T18:34:36.894 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice dbus.socket sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-01-15T18:34:36.896 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-01-15T18:34:36.925 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:36 +0000 (0:00:00.323) 0:03:27.486 ******** 2021-01-15T18:34:37.202 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-01-15T18:34:37.203 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-01-15T18:34:37.234 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:37 +0000 (0:00:00.308) 0:03:27.795 ******** 2021-01-15T18:34:37.924 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-01-15T18:34:37.926 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-01-15T18:34:37.956 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:37 +0000 (0:00:00.722) 0:03:28.518 ******** 2021-01-15T18:34:37.983 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-01-15T18:34:37.984 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.015 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.059) 0:03:28.577 ******** 2021-01-15T18:34:38.052 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-01-15T18:34:38.054 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.086 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.070) 0:03:28.647 ******** 2021-01-15T18:34:38.112 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-01-15T18:34:38.113 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.144 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.057) 0:03:28.705 ******** 2021-01-15T18:34:38.170 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-01-15T18:34:38.172 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.203 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.058) 0:03:28.764 ******** 2021-01-15T18:34:38.229 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-01-15T18:34:38.230 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.265 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.062) 0:03:28.827 ******** 2021-01-15T18:34:38.294 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-01-15T18:34:38.295 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.328 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.062) 0:03:28.890 ******** 2021-01-15T18:34:38.355 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-01-15T18:34:38.387 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.058) 0:03:28.949 ******** 2021-01-15T18:34:38.413 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-01-15T18:34:38.415 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.445 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.058) 0:03:29.007 ******** 2021-01-15T18:34:38.471 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-01-15T18:34:38.473 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.503 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.057) 0:03:29.065 ******** 2021-01-15T18:34:38.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-01-15T18:34:38.542 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.574 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.070) 0:03:29.135 ******** 2021-01-15T18:34:38.600 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-01-15T18:34:38.601 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.632 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.058) 0:03:29.193 ******** 2021-01-15T18:34:38.659 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-01-15T18:34:38.691 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.059) 0:03:29.253 ******** 2021-01-15T18:34:38.717 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.751 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.059) 0:03:29.313 ******** 2021-01-15T18:34:38.778 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-01-15T18:34:38.780 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.812 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.060) 0:03:29.373 ******** 2021-01-15T18:34:38.838 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-01-15T18:34:38.840 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.886 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.073) 0:03:29.447 ******** 2021-01-15T18:34:38.912 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-01-15T18:34:38.914 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:38.950 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:38 +0000 (0:00:00.063) 0:03:29.511 ******** 2021-01-15T18:34:38.976 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-01-15T18:34:38.978 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.009 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:29.570 ******** 2021-01-15T18:34:39.036 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-01-15T18:34:39.038 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.068 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:29.630 ******** 2021-01-15T18:34:39.095 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-01-15T18:34:39.097 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.128 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:29.689 ******** 2021-01-15T18:34:39.154 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-01-15T18:34:39.156 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.187 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:29.748 ******** 2021-01-15T18:34:39.213 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-01-15T18:34:39.215 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.247 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.060) 0:03:29.808 ******** 2021-01-15T18:34:39.274 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.308 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.061) 0:03:29.870 ******** 2021-01-15T18:34:39.335 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-01-15T18:34:39.336 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.366 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:29.928 ******** 2021-01-15T18:34:39.394 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-01-15T18:34:39.395 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.424 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.057) 0:03:29.986 ******** 2021-01-15T18:34:39.450 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-01-15T18:34:39.452 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.483 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.044 ******** 2021-01-15T18:34:39.509 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.542 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:30.104 ******** 2021-01-15T18:34:39.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-01-15T18:34:39.570 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.601 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.162 ******** 2021-01-15T18:34:39.627 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-01-15T18:34:39.629 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.660 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.221 ******** 2021-01-15T18:34:39.686 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-01-15T18:34:39.688 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.719 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.280 ******** 2021-01-15T18:34:39.745 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-01-15T18:34:39.746 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.777 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.339 ******** 2021-01-15T18:34:39.803 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-01-15T18:34:39.805 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.839 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.061) 0:03:30.400 ******** 2021-01-15T18:34:39.865 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-01-15T18:34:39.867 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.898 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.058) 0:03:30.459 ******** 2021-01-15T18:34:39.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:39.957 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:39 +0000 (0:00:00.059) 0:03:30.518 ******** 2021-01-15T18:34:39.984 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-01-15T18:34:39.985 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.017 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.060) 0:03:30.578 ******** 2021-01-15T18:34:40.044 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:34:40.045 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.077 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.059) 0:03:30.638 ******** 2021-01-15T18:34:40.104 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-01-15T18:34:40.106 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.136 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.058) 0:03:30.697 ******** 2021-01-15T18:34:40.162 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-01-15T18:34:40.164 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.195 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.059) 0:03:30.757 ******** 2021-01-15T18:34:40.222 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-01-15T18:34:40.223 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.254 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.058) 0:03:30.816 ******** 2021-01-15T18:34:40.280 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-01-15T18:34:40.282 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.314 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.059) 0:03:30.875 ******** 2021-01-15T18:34:40.340 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.372 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.058) 0:03:30.934 ******** 2021-01-15T18:34:40.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-01-15T18:34:40.401 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.432 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.059) 0:03:30.993 ******** 2021-01-15T18:34:40.458 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-01-15T18:34:40.460 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.490 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.057) 0:03:31.051 ******** 2021-01-15T18:34:40.516 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-01-15T18:34:40.517 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.550 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.060) 0:03:31.112 ******** 2021-01-15T18:34:40.577 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-01-15T18:34:40.579 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.626 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.075) 0:03:31.188 ******** 2021-01-15T18:34:40.663 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-01-15T18:34:40.665 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.705 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.078) 0:03:31.266 ******** 2021-01-15T18:34:40.732 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-01-15T18:34:40.734 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.773 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.068) 0:03:31.335 ******** 2021-01-15T18:34:40.811 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.845 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.071) 0:03:31.406 ******** 2021-01-15T18:34:40.871 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-01-15T18:34:40.873 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.914 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.068) 0:03:31.475 ******** 2021-01-15T18:34:40.943 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-01-15T18:34:40.945 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:40.976 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:40 +0000 (0:00:00.062) 0:03:31.537 ******** 2021-01-15T18:34:41.002 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-01-15T18:34:41.004 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.034 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.058) 0:03:31.595 ******** 2021-01-15T18:34:41.060 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-01-15T18:34:41.062 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.091 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.057) 0:03:31.652 ******** 2021-01-15T18:34:41.116 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-01-15T18:34:41.118 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.147 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:31.709 ******** 2021-01-15T18:34:41.174 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-01-15T18:34:41.175 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.206 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.058) 0:03:31.767 ******** 2021-01-15T18:34:41.232 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-01-15T18:34:41.234 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.265 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.059) 0:03:31.826 ******** 2021-01-15T18:34:41.292 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-01-15T18:34:41.294 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.324 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.059) 0:03:31.886 ******** 2021-01-15T18:34:41.350 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-01-15T18:34:41.353 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.383 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.058) 0:03:31.944 ******** 2021-01-15T18:34:41.409 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-01-15T18:34:41.411 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.441 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.057) 0:03:32.002 ******** 2021-01-15T18:34:41.466 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-01-15T18:34:41.468 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.498 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:32.059 ******** 2021-01-15T18:34:41.523 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.557 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.058) 0:03:32.118 ******** 2021-01-15T18:34:41.581 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-01-15T18:34:41.583 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.613 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:32.175 ******** 2021-01-15T18:34:41.639 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.669 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.055) 0:03:32.230 ******** 2021-01-15T18:34:41.694 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-01-15T18:34:41.696 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.725 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:32.287 ******** 2021-01-15T18:34:41.751 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-01-15T18:34:41.753 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.782 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:32.344 ******** 2021-01-15T18:34:41.808 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-01-15T18:34:41.810 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.839 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.056) 0:03:32.401 ******** 2021-01-15T18:34:41.865 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:34:41.867 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-01-15T18:34:41.897 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:41 +0000 (0:00:00.057) 0:03:32.458 ******** 2021-01-15T18:34:41.935 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-01-15T18:34:41.936 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.946 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-01-15T18:34:41.956 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.958 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.967 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.969 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:41.978 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-01-15T18:34:42.014 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.117) 0:03:32.575 ******** 2021-01-15T18:34:42.195 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-01-15T18:34:42.196 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003224", "end": "2021-01-15 18:34:42.169297", "rc": 0, "start": "2021-01-15 18:34:42.166073", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2021-01-15T18:34:42.231 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.217) 0:03:32.792 ******** 2021-01-15T18:34:42.257 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-01-15T18:34:42.259 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:42.292 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.060) 0:03:32.853 ******** 2021-01-15T18:34:42.318 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-01-15T18:34:42.319 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:42.349 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.057) 0:03:32.911 ******** 2021-01-15T18:34:42.386 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:42.418 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.068) 0:03:32.979 ******** 2021-01-15T18:34:42.443 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:42.474 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:42 +0000 (0:00:00.055) 0:03:33.035 ******** 2021-01-15T18:34:44.968 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-01-15T18:34:44.970 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2021-01-15T18:34:45.001 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:44 +0000 (0:00:02.527) 0:03:35.562 ******** 2021-01-15T18:34:45.037 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-01-15T18:34:45.039 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419957760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153310, 'block_used': 12941798, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153310, "block_size": 4096, "block_total": 240095108, "block_used": 12941798, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419957760, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-01-15T18:34:45.071 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:45 +0000 (0:00:00.069) 0:03:35.632 ******** 2021-01-15T18:34:45.262 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-01-15T18:34:45.264 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004985", "end": "2021-01-15 18:34:45.241920", "rc": 0, "start": "2021-01-15 18:34:45.236935", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:45.296 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:45 +0000 (0:00:00.224) 0:03:35.857 ******** 2021-01-15T18:34:45.333 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-01-15T18:34:45.334 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419957760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153310, 'block_used': 12941798, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153310, "block_size": 4096, "block_total": 240095108, "block_used": 12941798, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419957760, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:45.366 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:45 +0000 (0:00:00.070) 0:03:35.927 ******** 2021-01-15T18:34:45.392 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-01-15T18:34:45.426 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:45 +0000 (0:00:00.059) 0:03:35.987 ******** 2021-01-15T18:34:46.936 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-01-15T18:34:46.937 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTXrfbjScdj7w7Xqb0eVc8lNbHpsZuWRTA'], 'uuids': ['09ea903f-d63b-4317-ac9c-296aae90dfdb'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.306250", "end": "2021-01-15 18:34:46.910077", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTXrfbjScdj7w7Xqb0eVc8lNbHpsZuWRTA"], "labels": [], "masters": [], "uuids": ["09ea903f-d63b-4317-ac9c-296aae90dfdb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:45.603827", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:48.186 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-Aw4fO3-5dbZ-tAi2-svUn-4iAR-T4e0-tdis8p', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN', 'nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.065276", "end": "2021-01-15 18:34:48.157351", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Aw4fO3-5dbZ-tAi2-svUn-4iAR-T4e0-tdis8p", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN", "nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:47.092075", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-01-15T18:34:48.188 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:49.431 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTym0qyzcBJl7Q2L9WsjM9ZFOFkNn0fMxC'], 'uuids': ['d694e37b-2fb1-4ddd-8566-fee671a5faa0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.062386", "end": "2021-01-15 18:34:49.401792", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTym0qyzcBJl7Q2L9WsjM9ZFOFkNn0fMxC"], "labels": [], "masters": [], "uuids": ["d694e37b-2fb1-4ddd-8566-fee671a5faa0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:48.339406", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:50.673 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTKlJxVlg4LGmzvsUSCLu3YDLSjkawWw9f'], 'uuids': ['4479497d-7a7d-496d-a2a0-eed9da247e43'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.066865", "end": "2021-01-15 18:34:50.645742", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTKlJxVlg4LGmzvsUSCLu3YDLSjkawWw9f"], "labels": [], "masters": [], "uuids": ["4479497d-7a7d-496d-a2a0-eed9da247e43"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:49.578877", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:51.899 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTonOf0EUM9s0LKAhJcnEMc4OjCr8Gx8v0'], 'uuids': ['97395510-f18b-4a31-8cda-c25906ffaff6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.048366", "end": "2021-01-15 18:34:51.870271", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTonOf0EUM9s0LKAhJcnEMc4OjCr8Gx8v0"], "labels": [], "masters": [], "uuids": ["97395510-f18b-4a31-8cda-c25906ffaff6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:50.821905", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:51.911 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFK4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4', 'scsi-35000c50092924465', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4', 'wwn-0x5000c50092924465'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFK4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924465', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4-part1', 'scsi-35000c50092924465-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4-part1', 'wwn-0x5000c50092924465-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4", "scsi-35000c50092924465", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4", "wwn-0x5000c50092924465"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-01-15T18:34:51.912 INFO:teuthology.task.ansible.out:Z1W5XFK4-part1", "scsi-35000c50092924465-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4-part1", "wwn-0x5000c50092924465-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFK4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924465"}}, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:53.144 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTP0MidIYwNapwr7c2KyoiVY2c4ZQfgNam'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.055459", "end": "2021-01-15 18:34:53.112264", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTP0MidIYwNapwr7c2KyoiVY2c4ZQfgNam"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:52.056805", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T18:34:53.183 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:53 +0000 (0:00:07.757) 0:03:43.744 ******** 2021-01-15T18:34:53.560 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-01-15T18:34:53.562 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTXrfbjScdj7w7Xqb0eVc8lNbHpsZuWRTA'], 'uuids': ['09ea903f-d63b-4317-ac9c-296aae90dfdb'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140196", "end": "2021-01-15 18:34:53.521752", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTXrfbjScdj7w7Xqb0eVc8lNbHpsZuWRTA"], "labels": [], "masters": [], "uuids": ["09ea903f-d63b-4317-ac9c-296aae90dfdb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:53.381556", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137491 s, 839 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137491 s, 839 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:53.762 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-Aw4fO3-5dbZ-tAi2-svUn-4iAR-T4e0-tdis8p', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN', 'nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037750", "end": "2021-01-15 18:34:53.737434", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Aw4fO3-5dbZ-tAi2-svUn-4iAR-T4e0-tdis8p", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN", "nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:53.699684", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.035084 s, 3.3 GB/s", "stde 2021-01-15T18:34:53.764 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.035084 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:54.088 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTym0qyzcBJl7Q2L9WsjM9ZFOFkNn0fMxC'], 'uuids': ['d694e37b-2fb1-4ddd-8566-fee671a5faa0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.152848", "end": "2021-01-15 18:34:54.061599", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTym0qyzcBJl7Q2L9WsjM9ZFOFkNn0fMxC"], "labels": [], "masters": [], "uuids": ["d694e37b-2fb1-4ddd-8566-fee671a5faa0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:53.908751", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150239 s, 768 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150239 s, 768 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:54.403 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTKlJxVlg4LGmzvsUSCLu3YDLSjkawWw9f'], 'uuids': ['4479497d-7a7d-496d-a2a0-eed9da247e43'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.140299", "end": "2021-01-15 18:34:54.373526", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTKlJxVlg4LGmzvsUSCLu3YDLSjkawWw9f"], "labels": [], "masters": [], "uuids": ["4479497d-7a7d-496d-a2a0-eed9da247e43"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:54.233227", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137702 s, 838 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137702 s, 838 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:54.731 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTonOf0EUM9s0LKAhJcnEMc4OjCr8Gx8v0'], 'uuids': ['97395510-f18b-4a31-8cda-c25906ffaff6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140793", "end": "2021-01-15 18:34:54.699204", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTonOf0EUM9s0LKAhJcnEMc4OjCr8Gx8v0"], "labels": [], "masters": [], "uuids": ["97395510-f18b-4a31-8cda-c25906ffaff6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:54.558411", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138144 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138144 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:54.733 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFK4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4', 'scsi-35000c50092924465', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4', 'wwn-0x5000c50092924465'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFK4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924465', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4-part1', 'scsi-35000c50092924465-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4-part1', 'wwn-0x5000c50092924465-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFK4", "scsi-35000c50092924465", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4", "wwn-0x5000c50092924465"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFK4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-01-15T18:34:54.734 INFO:teuthology.task.ansible.out:Z1W5XFK4-part1", "scsi-35000c50092924465-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFK4-part1", "wwn-0x5000c50092924465-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFK4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924465"}}, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:55.049 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTP0MidIYwNapwr7c2KyoiVY2c4ZQfgNam'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.148207", "end": "2021-01-15 18:34:55.023406", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MedSgiE00uve2VyNG2YGoLlXedo5MYRTP0MidIYwNapwr7c2KyoiVY2c4ZQfgNam"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 18:34:54.875199", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14562 s, 792 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14562 s, 792 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:55.095 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:01.912) 0:03:45.657 ******** 2021-01-15T18:34:55.341 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-01-15T18:34:55.342 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.064430", "end": "2021-01-15 18:34:55.315433", "rc": 0, "start": "2021-01-15 18:34:55.251003", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:55.373 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.277) 0:03:45.934 ******** 2021-01-15T18:34:55.598 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-01-15T18:34:55.600 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047297", "end": "2021-01-15 18:34:55.576160", "rc": 0, "start": "2021-01-15 18:34:55.528863", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:34:55.632 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.259) 0:03:46.194 ******** 2021-01-15T18:34:55.659 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-01-15T18:34:55.694 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.061) 0:03:46.255 ******** 2021-01-15T18:34:55.720 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-01-15T18:34:55.722 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:55.757 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.063) 0:03:46.318 ******** 2021-01-15T18:34:55.783 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-01-15T18:34:55.785 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:55.816 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.059) 0:03:46.378 ******** 2021-01-15T18:34:55.843 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-01-15T18:34:55.845 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:55.882 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.065) 0:03:46.444 ******** 2021-01-15T18:34:55.909 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-01-15T18:34:55.910 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:55.945 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:55 +0000 (0:00:00.062) 0:03:46.506 ******** 2021-01-15T18:34:56.118 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-01-15T18:34:56.120 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T18:34:56.171 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.226) 0:03:46.733 ******** 2021-01-15T18:34:56.211 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-01-15T18:34:56.213 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419957760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153310, 'block_used': 12941798, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153310, "block_size": 4096, "block_total": 240095108, "block_used": 12941798, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419957760, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.256 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.084) 0:03:46.817 ******** 2021-01-15T18:34:56.285 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.339 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.083) 0:03:46.900 ******** 2021-01-15T18:34:56.368 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:34:56.370 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.414 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.075) 0:03:46.976 ******** 2021-01-15T18:34:56.444 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:34:56.445 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.478 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.063) 0:03:47.039 ******** 2021-01-15T18:34:56.504 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-01-15T18:34:56.506 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.538 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.059) 0:03:47.099 ******** 2021-01-15T18:34:56.564 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-01-15T18:34:56.565 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.601 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.063) 0:03:47.162 ******** 2021-01-15T18:34:56.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-01-15T18:34:56.632 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:34:56.669 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:56 +0000 (0:00:00.067) 0:03:47.230 ******** 2021-01-15T18:34:57.389 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-01-15T18:34:57.391 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-01-15T18:34:57.423 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:57 +0000 (0:00:00.754) 0:03:47.984 ******** 2021-01-15T18:34:58.026 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-01-15T18:34:58.028 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T18:34:58.348 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T18:34:58.650 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T18:34:58.970 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T18:34:59.283 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T18:34:59.319 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:59 +0000 (0:00:01.895) 0:03:49.880 ******** 2021-01-15T18:34:59.506 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-01-15T18:34:59.508 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-01-15T18:34:59.552 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:34:59 +0000 (0:00:00.233) 0:03:50.113 ******** 2021-01-15T18:34:59.756 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-01-15T18:34:59.758 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T18:34:59.933 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T18:35:00.111 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T18:35:00.279 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T18:35:00.290 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:00.336 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:00 +0000 (0:00:00.784) 0:03:50.897 ******** 2021-01-15T18:35:00.364 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:00.397 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:00 +0000 (0:00:00.061) 0:03:50.958 ******** 2021-01-15T18:35:00.424 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-01-15T18:35:00.425 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:00.457 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:00 +0000 (0:00:00.059) 0:03:51.018 ******** 2021-01-15T18:35:00.640 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-01-15T18:35:00.642 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T18:35:00.676 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:00 +0000 (0:00:00.219) 0:03:51.237 ******** 2021-01-15T18:35:00.703 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-01-15T18:35:00.705 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:00.749 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:00 +0000 (0:00:00.072) 0:03:51.310 ******** 2021-01-15T18:35:01.269 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-01-15T18:35:01.272 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true} 2021-01-15T18:35:01.307 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:01 +0000 (0:00:00.557) 0:03:51.868 ******** 2021-01-15T18:35:01.856 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-01-15T18:35:01.858 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-01-15T18:35:01.888 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:01 +0000 (0:00:00.581) 0:03:52.450 ******** 2021-01-15T18:35:01.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:01.950 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:01 +0000 (0:00:00.061) 0:03:52.511 ******** 2021-01-15T18:35:01.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-01-15T18:35:01.980 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.012 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.061) 0:03:52.573 ******** 2021-01-15T18:35:02.039 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.085 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.073) 0:03:52.646 ******** 2021-01-15T18:35:02.111 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-01-15T18:35:02.113 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.144 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.058) 0:03:52.705 ******** 2021-01-15T18:35:02.170 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-01-15T18:35:02.171 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.200 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.056) 0:03:52.762 ******** 2021-01-15T18:35:02.227 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-01-15T18:35:02.229 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.257 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.057) 0:03:52.819 ******** 2021-01-15T18:35:02.283 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-01-15T18:35:02.284 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.314 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.056) 0:03:52.876 ******** 2021-01-15T18:35:02.340 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-01-15T18:35:02.342 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.375 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.060) 0:03:52.936 ******** 2021-01-15T18:35:02.401 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-01-15T18:35:02.402 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.433 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.058) 0:03:52.994 ******** 2021-01-15T18:35:02.459 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-01-15T18:35:02.460 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.492 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.058) 0:03:53.053 ******** 2021-01-15T18:35:02.518 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.550 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.058) 0:03:53.111 ******** 2021-01-15T18:35:02.586 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.619 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.069) 0:03:53.180 ******** 2021-01-15T18:35:02.646 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-01-15T18:35:02.648 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.681 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.062) 0:03:53.242 ******** 2021-01-15T18:35:02.707 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-01-15T18:35:02.709 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.745 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.063) 0:03:53.306 ******** 2021-01-15T18:35:02.772 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:02.806 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:02 +0000 (0:00:00.060) 0:03:53.367 ******** 2021-01-15T18:35:03.258 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-01-15T18:35:03.260 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-01-15T18:35:03.292 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.486) 0:03:53.853 ******** 2021-01-15T18:35:03.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-01-15T18:35:03.580 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-01-15 18:30:23 UTC", "ActiveEnterTimestampMonotonic": "15740246", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service systemd-tmpfiles-setup.service sysinit.target system.slice ntpdate.service systemd-journald.socket -.mount ntpd.service tmp.mount basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-01-15 18:30:21 UTC", "AssertTimestampMonotonic": "14567053", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-01-15 18:30:21 UTC", "ConditionTimestampMonotonic": "14567016", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGro 2021-01-15T18:35:03.582 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "917", "ExecMainStartTimestamp": "Fri 2021-01-15 18:30:23 UTC", "ExecMainStartTimestampMonotonic": "15646470", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-01-15 18:30:21 UTC", "InactiveExitTimestampMonotonic": "14597265", "InvocationID": "b355fc45472c44b786652d3115e80aab", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-01-15T18:35:03.584 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "917", "MemoryAccounting": "yes", "MemoryCurrent": "3149824", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-01-15T18:35:03.585 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-01-15 18:30:23 UTC", "StateChangeTimestampMonotonic": "15740246", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-01-15 18:30:23 UTC", "WatchdogTimestampMonotonic": "15646481", "WatchdogUSec": "0"}} 2021-01-15T18:35:03.615 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.323) 0:03:54.176 ******** 2021-01-15T18:35:03.642 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-01-15T18:35:03.692 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.076) 0:03:54.253 ******** 2021-01-15T18:35:03.721 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-01-15T18:35:03.756 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.063) 0:03:54.317 ******** 2021-01-15T18:35:03.793 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:03.828 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.072) 0:03:54.389 ******** 2021-01-15T18:35:03.854 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-01-15T18:35:03.856 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:03.887 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.059) 0:03:54.449 ******** 2021-01-15T18:35:03.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:03.947 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:03 +0000 (0:00:00.060) 0:03:54.509 ******** 2021-01-15T18:35:04.131 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-01-15T18:35:04.165 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:04 +0000 (0:00:00.217) 0:03:54.727 ******** 2021-01-15T18:35:04.615 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-01-15T18:35:04.617 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-01-15T18:35:04.649 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:04 +0000 (0:00:00.483) 0:03:55.210 ******** 2021-01-15T18:35:05.942 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-01-15T18:35:05.944 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-01-15T18:35:05.986 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:05 +0000 (0:00:01.337) 0:03:56.548 ******** 2021-01-15T18:35:06.014 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-01-15T18:35:06.016 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:06.051 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.064) 0:03:56.612 ******** 2021-01-15T18:35:06.077 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-01-15T18:35:06.079 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-01-15T18:35:06.111 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.059) 0:03:56.672 ******** 2021-01-15T18:35:06.306 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-01-15T18:35:06.308 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-01-15T18:35:06.338 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.227) 0:03:56.899 ******** 2021-01-15T18:35:06.365 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-01-15T18:35:06.367 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:06.398 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.060) 0:03:56.960 ******** 2021-01-15T18:35:06.424 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-01-15T18:35:06.426 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-01-15T18:35:06.457 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.059) 0:03:57.019 ******** 2021-01-15T18:35:06.483 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-01-15T18:35:06.485 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:06.516 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.058) 0:03:57.077 ******** 2021-01-15T18:35:06.697 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-01-15T18:35:06.699 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-01-15T18:35:06.732 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.215) 0:03:57.293 ******** 2021-01-15T18:35:06.914 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-01-15T18:35:06.917 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-01-15T18:35:06.948 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:06 +0000 (0:00:00.216) 0:03:57.509 ******** 2021-01-15T18:35:06.974 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-01-15T18:35:06.975 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.006 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.058) 0:03:57.567 ******** 2021-01-15T18:35:07.043 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-01-15T18:35:07.044 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.046 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.056 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.057 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.089 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.083) 0:03:57.651 ******** 2021-01-15T18:35:07.125 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-01-15T18:35:07.126 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.128 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.138 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.139 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.176 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.086) 0:03:57.737 ******** 2021-01-15T18:35:07.202 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T18:35:07.204 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.237 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.060) 0:03:57.798 ******** 2021-01-15T18:35:07.262 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-01-15T18:35:07.264 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.295 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.058) 0:03:57.856 ******** 2021-01-15T18:35:07.320 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-01-15T18:35:07.322 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.354 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.059) 0:03:57.916 ******** 2021-01-15T18:35:07.383 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-01-15T18:35:07.385 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.415 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.060) 0:03:57.976 ******** 2021-01-15T18:35:07.587 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-01-15T18:35:07.589 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.620 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.205) 0:03:58.182 ******** 2021-01-15T18:35:07.646 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-01-15T18:35:07.648 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.681 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.061) 0:03:58.243 ******** 2021-01-15T18:35:07.708 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-01-15T18:35:07.709 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.742 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.060) 0:03:58.304 ******** 2021-01-15T18:35:07.769 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-01-15T18:35:07.771 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:07.802 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:07 +0000 (0:00:00.060) 0:03:58.364 ******** 2021-01-15T18:35:09.013 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-01-15T18:35:09.015 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006266", "end": "2021-01-15 18:35:08.980059", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-01-15 18:35:07.973793", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:35:09.190 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003044", "end": "2021-01-15 18:35:09.162585", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-01-15 18:35:09.159541", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:35:09.357 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002826", "end": "2021-01-15 18:35:09.328288", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-01-15 18:35:09.325462", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-01-15T18:35:09.534 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015941", "end": "2021-01-15 18:35:09.508352", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-01-15 18:35:09.492411", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:35:09.569 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:09 +0000 (0:00:01.765) 0:04:00.130 ******** 2021-01-15T18:35:09.595 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-01-15T18:35:09.627 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:09 +0000 (0:00:00.058) 0:04:00.188 ******** 2021-01-15T18:35:09.810 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-01-15T18:35:09.812 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-01-15T18:35:09.911 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-01-15T18:35:09.943 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:09 +0000 (0:00:00.315) 0:04:00.504 ******** 2021-01-15T18:35:10.571 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi148.front.sepia.ceph.com] 2021-01-15T18:35:10.642 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:10 +0000 (0:00:00.699) 0:04:01.204 ******** 2021-01-15T18:35:10.669 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:10.703 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:10 +0000 (0:00:00.060) 0:04:01.265 ******** 2021-01-15T18:35:10.729 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T18:35:10.730 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-01-15T18:35:10.764 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:10 +0000 (0:00:00.060) 0:04:01.325 ******** 2021-01-15T18:35:10.801 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-01-15T18:35:10.803 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-01-15T18:35:10.841 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:10 +0000 (0:00:00.077) 0:04:01.403 ******** 2021-01-15T18:35:13.352 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-01-15T18:35:13.354 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: podman", "Installed: podman-docker"]} 2021-01-15T18:35:13.386 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:13 +0000 (0:00:02.544) 0:04:03.947 ******** 2021-01-15T18:35:13.411 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T18:35:13.413 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-01-15T18:35:13.444 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:13 +0000 (0:00:00.058) 0:04:04.006 ******** 2021-01-15T18:35:13.471 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T18:35:13.473 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:13.506 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:13 +0000 (0:00:00.061) 0:04:04.067 ******** 2021-01-15T18:35:13.707 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-01-15T18:35:13.709 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T18:35:13.893 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T18:35:13.931 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:13 +0000 (0:00:00.425) 0:04:04.493 ******** 2021-01-15T18:35:14.420 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-01-15T18:35:14.422 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "ad23d23287af05eb6dfc6847a1957988954f5ccf", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "97bf7a9f95b52e04e40f96163cd1717c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2134, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610735713.967215-17351334672697/source", "state": "file", "uid": 0} 2021-01-15T18:35:14.878 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "ad23d23287af05eb6dfc6847a1957988954f5ccf", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "97bf7a9f95b52e04e40f96163cd1717c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2134, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610735714.4245665-147525535870242/source", "state": "file", "uid": 0} 2021-01-15T18:35:14.914 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:14 +0000 (0:00:00.982) 0:04:05.475 ******** 2021-01-15T18:35:17.997 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-01-15T18:35:17.998 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-lpy8utcz-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-lpy8utcz-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-01-15T18:35:18.000 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-01-15T18:35:18.050 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:18 +0000 (0:00:03.136) 0:04:08.612 ******** 2021-01-15T18:35:18.078 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-01-15T18:35:18.080 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:18.115 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:18 +0000 (0:00:00.064) 0:04:08.676 ******** 2021-01-15T18:35:18.152 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-01-15T18:35:18.154 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:18.186 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:18 +0000 (0:00:00.071) 0:04:08.748 ******** 2021-01-15T18:35:18.524 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-01-15T18:35:18.525 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.156133", "end": "2021-01-15 18:35:18.506308", "rc": 0, "start": "2021-01-15 18:35:18.350175", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T18:35:18.559 INFO:teuthology.task.ansible.out:Friday 15 January 2021 18:35:18 +0000 (0:00:00.372) 0:04:09.120 ******** 2021-01-15T18:35:18.585 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-01-15T18:35:18.586 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T18:35:18.678 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-01-15T18:35:18.682 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-01-15T18:35:18.685 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-01-15T18:35:18.687 INFO:teuthology.task.ansible.out:PLAY RECAP ********************************************************************* smithi148.front.sepia.ceph.com : ok=136 changed=43 unreachable=0 failed=0 skipped=173 rescued=0 ignored=4 Friday 15 January 2021 18:35:18 +0000 (0:00:00.126) 0:04:09.247 ******** =============================================================================== users : Update authorized_keys using the keys repo --------------------- 31.44s users : Create all admin users with sudo access. ----------------------- 29.86s common : nrpe - Remove SELinux policy package -------------------------- 16.41s common : nrpe - Load SELinux policy package ---------------------------- 16.35s common : Register with subscription-manager. --------------------------- 15.89s testnode : Install packages -------------------------------------------- 11.83s users : Remove revoked users -------------------------------------------- 9.98s testnode : Zap all non-root disks --------------------------------------- 7.76s testnode : Ensure ceph packages are not present. ------------------------ 6.01s testnode : Ensure ceph dependency packages are not present. ------------- 5.49s common : Enable necessary rhsm repos ------------------------------------ 3.14s container-host : Install registries-conf-ctl ---------------------------- 3.14s common : Install Kerberos Packages (RedHat) ----------------------------- 2.91s testnode : Remove packages ---------------------------------------------- 2.61s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 2.61s common : Get list of enabled RHSM repos --------------------------------- 2.60s common : nrpe - Ensure SELinux policy is up to date --------------------- 2.58s common : nrpe - Install SELinux dependencies ---------------------------- 2.57s container-host : Install container packages ----------------------------- 2.54s testnode : Install epel packages ---------------------------------------- 2.54s 2021-01-15T18:35:18.985 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2021-01-15T18:35:18.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T18:35:18.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T18:35:19.056 DEBUG:teuthology.orchestra.run.smithi148:> true 2021-01-15T18:35:19.150 INFO:teuthology.run_tasks:Running task clock... 2021-01-15T18:35:19.162 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-01-15T18:35:19.163 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-01-15T18:35:19.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-01-15T18:35:19.219 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-01-15T18:35:19.233 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-01-15T18:35:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2021-01-15T18:35:19.278 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2021-01-15T18:35:19.292 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-01-15T18:35:19.305 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-01-15T18:35:19.360 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2021-01-15T18:35:19.363 INFO:teuthology.orchestra.run.smithi148.stdout:210 Number of sources = 4 2021-01-15T18:35:19.363 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-01-15T18:35:19.363 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2021-01-15T18:35:19.364 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T18:35:19.364 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T18:35:19.364 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T18:35:19.364 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T18:35:19.365 INFO:teuthology.run_tasks:Running task install... 2021-01-15T18:35:19.375 DEBUG:teuthology.task.install:project ceph 2021-01-15T18:35:19.376 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'}} 2021-01-15T18:35:19.376 DEBUG:teuthology.task.install:config {'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'} 2021-01-15T18:35:19.376 INFO:teuthology.task.install:Using flavor: basic 2021-01-15T18:35:19.382 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-01-15T18:35:19.382 INFO:teuthology.task.install:extra packages: [] 2021-01-15T18:35:19.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2021-01-15T18:35:19.438 INFO:teuthology.orchestra.run.smithi148.stdout:RedHatEnterprise 2021-01-15T18:35:19.439 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-01-15T18:35:19.439 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3', 'tag': None, 'wait_for_package': False} 2021-01-15T18:35:19.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2021-01-15T18:35:19.511 INFO:teuthology.orchestra.run.smithi148.stdout:RedHatEnterprise 2021-01-15T18:35:19.512 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-01-15T18:35:19.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T18:35:19.675 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/ 2021-01-15T18:35:19.675 INFO:teuthology.task.install.rpm:Package version is 16.0.0-9031.g892c51dd 2021-01-15T18:35:19.799 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-01-15T18:35:19.800 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:35:19.800 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-01-15T18:35:19.824 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-01-15T18:35:19.825 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-01-15T18:35:19.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-01-15T18:35:19.963 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-01-15T18:35:20.027 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2021-01-15T18:35:20.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2021-01-15T18:35:20.334 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:35:20.576 INFO:teuthology.orchestra.run.smithi148.stdout:43 files removed 2021-01-15T18:35:20.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2021-01-15T18:35:20.876 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:35:21.513 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 294 kB/s | 73 kB 00:00 2021-01-15T18:35:21.747 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 74 kB/s | 15 kB 00:00 2021-01-15T18:35:21.973 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 7.0 kB/s | 1.4 kB 00:00 2021-01-15T18:35:22.611 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 14 MB/s | 8.8 MB 00:00 2021-01-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 4.5 MB/s | 23 kB 00:00 2021-01-15T18:35:24.611 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 122 MB/s | 27 MB 00:00 2021-01-15T18:35:28.467 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 114 MB/s | 25 MB 00:00 2021-01-15T18:35:34.700 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:35:34.707 INFO:teuthology.orchestra.run.smithi148.stdout:==================================================================================================== 2021-01-15T18:35:34.708 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:35:34.708 INFO:teuthology.orchestra.run.smithi148.stdout:==================================================================================================== 2021-01-15T18:35:34.708 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:35:34.708 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 12 M 2021-01-15T18:35:34.709 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2021-01-15T18:35:34.709 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.7 M 2021-01-15T18:35:34.709 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.2 M 2021-01-15T18:35:34.709 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:35:34.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 6.0 M 2021-01-15T18:35:34.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 24 M 2021-01-15T18:35:34.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 25 k 2021-01-15T18:35:34.710 INFO:teuthology.orchestra.run.smithi148.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-01-15T18:35:34.711 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2021-01-15T18:35:34.711 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-01-15T18:35:34.711 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 793 k 2021-01-15T18:35:34.711 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-01-15T18:35:34.712 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-01-15T18:35:34.712 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 486 k 2021-01-15T18:35:34.712 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-01-15T18:35:34.712 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 4.0 M 2021-01-15T18:35:34.712 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-01-15T18:35:34.713 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-01-15T18:35:34.713 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 40 k 2021-01-15T18:35:34.713 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 77 k 2021-01-15T18:35:34.713 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 186 k 2021-01-15T18:35:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-01-15T18:35:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 387 k 2021-01-15T18:35:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 368 k 2021-01-15T18:35:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 114 k 2021-01-15T18:35:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:35:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:35:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:==================================================================================================== 2021-01-15T18:35:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:Install 22 Packages 2021-01-15T18:35:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2021-01-15T18:35:34.716 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:35:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 57 M 2021-01-15T18:35:34.717 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:35:35.393 INFO:teuthology.orchestra.run.smithi148.stdout:(1/24): ceph-base-16.0.0-9031.g892c51dd.el8.x86 8.8 MB/s | 6.0 MB 00:00 2021-01-15T18:35:35.430 INFO:teuthology.orchestra.run.smithi148.stdout:(2/24): ceph-selinux-16.0.0-9031.g892c51dd.el8. 660 kB/s | 25 kB 00:00 2021-01-15T18:35:35.533 INFO:teuthology.orchestra.run.smithi148.stdout:(3/24): libcephfs2-16.0.0-9031.g892c51dd.el8.x8 7.5 MB/s | 793 kB 00:00 2021-01-15T18:35:35.589 INFO:teuthology.orchestra.run.smithi148.stdout:(4/24): libradosstriper1-16.0.0-9031.g892c51dd. 8.5 MB/s | 486 kB 00:00 2021-01-15T18:35:35.969 INFO:teuthology.orchestra.run.smithi148.stdout:(5/24): librgw2-16.0.0-9031.g892c51dd.el8.x86_6 10 MB/s | 4.0 MB 00:00 2021-01-15T18:35:36.013 INFO:teuthology.orchestra.run.smithi148.stdout:(6/24): python3-ceph-argparse-16.0.0-9031.g892c 927 kB/s | 40 kB 00:00 2021-01-15T18:35:36.057 INFO:teuthology.orchestra.run.smithi148.stdout:(7/24): python3-ceph-common-16.0.0-9031.g892c51 1.7 MB/s | 77 kB 00:00 2021-01-15T18:35:36.107 INFO:teuthology.orchestra.run.smithi148.stdout:(8/24): python3-cephfs-16.0.0-9031.g892c51dd.el 3.9 MB/s | 186 kB 00:00 2021-01-15T18:35:36.160 INFO:teuthology.orchestra.run.smithi148.stdout:(9/24): python3-rados-16.0.0-9031.g892c51dd.el8 6.8 MB/s | 387 kB 00:00 2021-01-15T18:35:36.215 INFO:teuthology.orchestra.run.smithi148.stdout:(10/24): python3-rbd-16.0.0-9031.g892c51dd.el8. 6.5 MB/s | 368 kB 00:00 2021-01-15T18:35:36.261 INFO:teuthology.orchestra.run.smithi148.stdout:(11/24): python3-rgw-16.0.0-9031.g892c51dd.el8. 2.5 MB/s | 114 kB 00:00 2021-01-15T18:35:36.343 INFO:teuthology.orchestra.run.smithi148.stdout:(12/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2021-01-15T18:35:36.391 INFO:teuthology.orchestra.run.smithi148.stdout:(13/24): gperftools-libs-2.7-6.el8.x86_64.rpm 6.3 MB/s | 315 kB 00:00 2021-01-15T18:35:36.424 INFO:teuthology.orchestra.run.smithi148.stdout:(14/24): leveldb-1.22-1.el8.x86_64.rpm 5.2 MB/s | 181 kB 00:00 2021-01-15T18:35:36.441 INFO:teuthology.orchestra.run.smithi148.stdout:(15/24): liboath-2.6.2-3.el8.x86_64.rpm 3.4 MB/s | 59 kB 00:00 2021-01-15T18:35:36.458 INFO:teuthology.orchestra.run.smithi148.stdout:(16/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2021-01-15T18:35:36.499 INFO:teuthology.orchestra.run.smithi148.stdout:(17/24): ceph-radosgw-16.0.0-9031.g892c51dd.el8 6.9 MB/s | 12 MB 00:01 2021-01-15T18:35:36.530 INFO:teuthology.orchestra.run.smithi148.stdout:(18/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 653 kB/s | 47 kB 00:00 2021-01-15T18:35:36.537 INFO:teuthology.orchestra.run.smithi148.stdout:(19/24): python3-prettytable-0.7.2-14.el8.noarc 1.2 MB/s | 44 kB 00:00 2021-01-15T18:35:36.556 INFO:teuthology.orchestra.run.smithi148.stdout:(20/24): librdkafka-0.11.4-1.el8.x86_64.rpm 13 MB/s | 353 kB 00:00 2021-01-15T18:35:36.722 INFO:teuthology.orchestra.run.smithi148.stdout:(21/24): ceph-common-16.0.0-9031.g892c51dd.el8. 12 MB/s | 24 MB 00:02 2021-01-15T18:35:36.724 INFO:teuthology.orchestra.run.smithi148.stdout:(22/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2021-01-15T18:35:36.863 INFO:teuthology.orchestra.run.smithi148.stdout:(23/24): librados2-16.0.0-9031.g892c51dd.el8.x8 12 MB/s | 3.7 MB 00:00 2021-01-15T18:35:36.919 INFO:teuthology.orchestra.run.smithi148.stdout:(24/24): librbd1-16.0.0-9031.g892c51dd.el8.x86_ 16 MB/s | 3.2 MB 00:00 2021-01-15T18:35:36.922 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:35:36.923 INFO:teuthology.orchestra.run.smithi148.stdout:Total 26 MB/s | 57 MB 00:02 2021-01-15T18:35:36.926 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:35:36.971 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:35:36.971 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:35:37.417 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:35:37.421 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:35:38.562 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:35:38.763 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-01-15T18:35:38.883 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-01-15T18:35:39.338 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-01-15T18:35:39.475 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 3/26 2021-01-15T18:35:39.571 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 3/26 2021-01-15T18:35:39.749 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 4/26 2021-01-15T18:35:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/26 2021-01-15T18:35:40.133 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 6/26 2021-01-15T18:35:40.485 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 6/26 2021-01-15T18:35:40.691 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T18:35:40.821 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T18:35:40.941 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/26 2021-01-15T18:35:41.024 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/26 2021-01-15T18:35:41.544 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 9/26 2021-01-15T18:35:41.674 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T18:35:41.757 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T18:35:41.904 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-01-15T18:35:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8. 12/26 2021-01-15T18:35:42.194 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 13/26 2021-01-15T18:35:42.384 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 14/26 2021-01-15T18:35:42.607 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 15/26 2021-01-15T18:35:42.724 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 16/26 2021-01-15T18:35:42.799 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 16/26 2021-01-15T18:35:42.976 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/26 2021-01-15T18:35:43.128 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 18/26 2021-01-15T18:35:43.353 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 19/26 2021-01-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x8 20/26 2021-01-15T18:35:47.613 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T18:35:47.791 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T18:35:48.524 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T18:35:48.754 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 22/26 2021-01-15T18:35:48.840 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 22/26 2021-01-15T18:35:48.973 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T18:36:05.941 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T18:36:05.941 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2021-01-15T18:36:05.942 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2021-01-15T18:36:05.942 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2021-01-15T18:36:05.942 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /run 2021-01-15T18:36:05.942 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2021-01-15T18:36:05.942 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2021-01-15T18:36:05.943 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2021-01-15T18:36:05.943 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2021-01-15T18:36:05.943 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2021-01-15T18:36:05.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:07.254 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 24/26 2021-01-15T18:36:07.314 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 24/26 2021-01-15T18:36:07.315 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-01-15T18:36:07.506 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-01-15T18:36:07.506 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T18:36:08.218 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T18:36:08.219 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 1/26 2021-01-15T18:36:08.219 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2/26 2021-01-15T18:36:08.219 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 3/26 2021-01-15T18:36:08.219 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 4/26 2021-01-15T18:36:08.220 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 5/26 2021-01-15T18:36:08.220 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 6/26 2021-01-15T18:36:08.220 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T18:36:08.220 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8. 8/26 2021-01-15T18:36:08.220 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x8 9/26 2021-01-15T18:36:08.221 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T18:36:08.221 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 11/26 2021-01-15T18:36:08.221 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 12/26 2021-01-15T18:36:08.221 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 13/26 2021-01-15T18:36:08.221 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 14/26 2021-01-15T18:36:08.222 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 15/26 2021-01-15T18:36:08.222 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : leveldb-1.22-1.el8.x86_64 16/26 2021-01-15T18:36:08.222 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 17/26 2021-01-15T18:36:08.222 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 18/26 2021-01-15T18:36:08.222 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 19/26 2021-01-15T18:36:08.223 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/26 2021-01-15T18:36:08.223 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/26 2021-01-15T18:36:08.223 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 22/26 2021-01-15T18:36:08.223 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T18:36:08.223 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-01-15T18:36:08.224 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 25/26 2021-01-15T18:36:09.066 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T18:36:09.066 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:36:10.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:10.867 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2021-01-15T18:36:10.868 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.868 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:10.868 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:36:10.869 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.869 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.869 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.869 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.869 INFO:teuthology.orchestra.run.smithi148.stdout: fmt-6.2.1-1.el8.x86_64 2021-01-15T18:36:10.870 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.7-6.el8.x86_64 2021-01-15T18:36:10.870 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb-1.22-1.el8.x86_64 2021-01-15T18:36:10.870 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.870 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.2-3.el8.x86_64 2021-01-15T18:36:10.870 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-01-15T18:36:10.871 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.871 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-01-15T18:36:10.871 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.871 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.3.1-3.el8.x86_64 2021-01-15T18:36:10.871 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-01-15T18:36:10.872 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.872 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.872 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.872 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-01-15T18:36:10.873 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.873 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.873 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:10.873 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:10.873 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:36:10.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2021-01-15T18:36:11.236 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:36:11.715 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-01-15T18:36:11.825 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-01-15T18:36:13.334 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:36:13.335 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2021-01-15T18:36:13.335 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:36:13.336 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2021-01-15T18:36:13.336 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:36:13.336 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 49 M 2021-01-15T18:36:13.336 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:36:13.337 INFO:teuthology.orchestra.run.smithi148.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-01-15T18:36:13.337 INFO:teuthology.orchestra.run.smithi148.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-01-15T18:36:13.337 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-01-15T18:36:13.337 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-01-15T18:36:13.337 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-01-15T18:36:13.338 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:13.338 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:36:13.338 INFO:teuthology.orchestra.run.smithi148.stdout:========================================================================================= 2021-01-15T18:36:13.338 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2021-01-15T18:36:13.339 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:13.339 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 50 M 2021-01-15T18:36:13.339 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 218 M 2021-01-15T18:36:13.340 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:36:13.369 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): libxslt-1.1.32-5.el8.x86_64.rpm 7.8 MB/s | 250 kB 00:00 2021-01-15T18:36:13.390 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 7.7 MB/s | 161 kB 00:00 2021-01-15T18:36:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 14 MB/s | 302 kB 00:00 2021-01-15T18:36:13.419 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 844 kB/s | 69 kB 00:00 2021-01-15T18:36:13.430 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.0 MB/s | 187 kB 00:00 2021-01-15T18:36:15.750 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): ceph-test-16.0.0-9031.g892c51dd.el8.x86_ 20 MB/s | 49 MB 00:02 2021-01-15T18:36:15.753 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:36:15.753 INFO:teuthology.orchestra.run.smithi148.stdout:Total 21 MB/s | 50 MB 00:02 2021-01-15T18:36:15.757 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:36:15.779 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:36:15.780 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:36:16.082 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:36:16.143 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:36:16.853 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:36:16.969 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2021-01-15T18:36:17.040 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2021-01-15T18:36:17.231 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : jq-1.5-12.el8.x86_64 2/6 2021-01-15T18:36:17.424 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-01-15T18:36:17.663 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 4/6 2021-01-15T18:36:22.400 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 5/6 2021-01-15T18:36:22.699 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 6/6 2021-01-15T18:36:22.918 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 6/6 2021-01-15T18:36:22.918 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 1/6 2021-01-15T18:36:22.918 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-01-15T18:36:22.919 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 3/6 2021-01-15T18:36:22.919 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-01-15T18:36:22.919 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/6 2021-01-15T18:36:23.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 6/6 2021-01-15T18:36:23.443 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:36:25.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:25.360 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:36:25.360 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 jq-1.5-12.el8.x86_64 2021-01-15T18:36:25.360 INFO:teuthology.orchestra.run.smithi148.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-01-15T18:36:25.360 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-01-15T18:36:25.361 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:25.361 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:36:25.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2021-01-15T18:36:25.718 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:36:26.218 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-01-15T18:36:26.328 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-01-15T18:36:27.812 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:36:27.816 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================= 2021-01-15T18:36:27.816 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:36:27.816 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================= 2021-01-15T18:36:27.816 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:36:27.817 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 6.4 k 2021-01-15T18:36:27.817 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:36:27.817 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 2.4 M 2021-01-15T18:36:27.817 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 1.5 M 2021-01-15T18:36:27.817 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 200 k 2021-01-15T18:36:27.818 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 4.9 M 2021-01-15T18:36:27.818 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 18 M 2021-01-15T18:36:27.818 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-01-15T18:36:27.819 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-01-15T18:36:27.819 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-01-15T18:36:27.819 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-01-15T18:36:27.819 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.5.1-1.el8 epel 171 k 2021-01-15T18:36:27.820 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-01-15T18:36:27.820 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-01-15T18:36:27.820 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-01-15T18:36:27.820 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-01-15T18:36:27.821 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-01-15T18:36:27.821 INFO:teuthology.orchestra.run.smithi148.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-01-15T18:36:27.821 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-01-15T18:36:27.822 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-01-15T18:36:27.822 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-01-15T18:36:27.822 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-01-15T18:36:27.823 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-01-15T18:36:27.823 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-01-15T18:36:27.823 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-01-15T18:36:27.824 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-01-15T18:36:27.824 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-01-15T18:36:27.825 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-01-15T18:36:27.825 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-01-15T18:36:27.825 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-01-15T18:36:27.826 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-01-15T18:36:27.826 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-01-15T18:36:27.827 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-01-15T18:36:27.827 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-01-15T18:36:27.827 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-01-15T18:36:27.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-01-15T18:36:27.828 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2021-01-15T18:36:27.828 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-01-15T18:36:27.829 INFO:teuthology.orchestra.run.smithi148.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-01-15T18:36:27.830 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:27.830 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:36:27.831 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================= 2021-01-15T18:36:27.831 INFO:teuthology.orchestra.run.smithi148.stdout:Install 37 Packages 2021-01-15T18:36:27.831 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:27.833 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 33 M 2021-01-15T18:36:27.833 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 117 M 2021-01-15T18:36:27.833 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:36:27.924 INFO:teuthology.orchestra.run.smithi148.stdout:(1/37): ceph-16.0.0-9031.g892c51dd.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-01-15T18:36:28.128 INFO:teuthology.orchestra.run.smithi148.stdout:(2/37): ceph-mgr-16.0.0-9031.g892c51dd.el8.x86_ 5.0 MB/s | 1.5 MB 00:00 2021-01-15T18:36:28.180 INFO:teuthology.orchestra.run.smithi148.stdout:(3/37): ceph-mds-16.0.0-9031.g892c51dd.el8.x86_ 6.7 MB/s | 2.4 MB 00:00 2021-01-15T18:36:28.241 INFO:teuthology.orchestra.run.smithi148.stdout:(4/37): ceph-mgr-modules-core-16.0.0-9031.g892c 3.2 MB/s | 200 kB 00:00 2021-01-15T18:36:28.307 INFO:teuthology.orchestra.run.smithi148.stdout:(5/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 661 kB/s | 44 kB 00:00 2021-01-15T18:36:28.355 INFO:teuthology.orchestra.run.smithi148.stdout:(6/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.7 MB/s | 185 kB 00:00 2021-01-15T18:36:28.388 INFO:teuthology.orchestra.run.smithi148.stdout:(7/37): python3-cheroot-8.5.1-1.el8.noarch.rpm 5.2 MB/s | 171 kB 00:00 2021-01-15T18:36:28.439 INFO:teuthology.orchestra.run.smithi148.stdout:(8/37): python3-cherrypy-18.4.0-1.el8.noarch.rp 7.6 MB/s | 384 kB 00:00 2021-01-15T18:36:28.457 INFO:teuthology.orchestra.run.smithi148.stdout:(9/37): ceph-mon-16.0.0-9031.g892c51dd.el8.x86_ 9.2 MB/s | 4.9 MB 00:00 2021-01-15T18:36:28.458 INFO:teuthology.orchestra.run.smithi148.stdout:(10/37): python3-cssselect-0.9.2-10.el8.noarch. 1.9 MB/s | 40 kB 00:00 2021-01-15T18:36:28.474 INFO:teuthology.orchestra.run.smithi148.stdout:(11/37): python3-jaraco-functools-2.0-4.el8.noa 1.1 MB/s | 18 kB 00:00 2021-01-15T18:36:28.492 INFO:teuthology.orchestra.run.smithi148.stdout:(12/37): python3-logutils-0.3.5-11.el8.noarch.r 2.8 MB/s | 49 kB 00:00 2021-01-15T18:36:28.497 INFO:teuthology.orchestra.run.smithi148.stdout:(13/37): python3-jaraco-6.2-6.el8.noarch.rpm 282 kB/s | 11 kB 00:00 2021-01-15T18:36:28.511 INFO:teuthology.orchestra.run.smithi148.stdout:(14/37): python3-more-itertools-7.2.0-3.el8.noa 3.5 MB/s | 59 kB 00:00 2021-01-15T18:36:28.525 INFO:teuthology.orchestra.run.smithi148.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 942 kB/s | 16 kB 00:00 2021-01-15T18:36:28.542 INFO:teuthology.orchestra.run.smithi148.stdout:(16/37): python3-simplegeneric-0.8.1-17.el8.noa 1.1 MB/s | 19 kB 00:00 2021-01-15T18:36:28.558 INFO:teuthology.orchestra.run.smithi148.stdout:(17/37): python3-singledispatch-3.4.0.3-18.el8. 1.4 MB/s | 24 kB 00:00 2021-01-15T18:36:28.578 INFO:teuthology.orchestra.run.smithi148.stdout:(18/37): python3-pecan-1.3.2-9.el8.noarch.rpm 3.5 MB/s | 283 kB 00:00 2021-01-15T18:36:28.578 INFO:teuthology.orchestra.run.smithi148.stdout:(19/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.7 MB/s | 29 kB 00:00 2021-01-15T18:36:28.595 INFO:teuthology.orchestra.run.smithi148.stdout:(20/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-01-15T18:36:28.608 INFO:teuthology.orchestra.run.smithi148.stdout:(21/37): python3-waitress-1.2.1-2.el8.1.noarch. 7.1 MB/s | 241 kB 00:00 2021-01-15T18:36:28.625 INFO:teuthology.orchestra.run.smithi148.stdout:(22/37): python3-webtest-2.0.33-1.el8.noarch.rp 4.9 MB/s | 85 kB 00:00 2021-01-15T18:36:28.634 INFO:teuthology.orchestra.run.smithi148.stdout:(23/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.2 MB/s | 251 kB 00:00 2021-01-15T18:36:28.643 INFO:teuthology.orchestra.run.smithi148.stdout:(24/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.3 MB/s | 23 kB 00:00 2021-01-15T18:36:28.675 INFO:teuthology.orchestra.run.smithi148.stdout:(25/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.8 MB/s | 238 kB 00:00 2021-01-15T18:36:28.683 INFO:teuthology.orchestra.run.smithi148.stdout:(26/37): python3-asn1crypto-0.24.0-3.el8.noarch 4.5 MB/s | 181 kB 00:00 2021-01-15T18:36:28.697 INFO:teuthology.orchestra.run.smithi148.stdout:(27/37): python3-pycparser-2.14-14.el8.noarch.r 4.9 MB/s | 109 kB 00:00 2021-01-15T18:36:28.717 INFO:teuthology.orchestra.run.smithi148.stdout:(28/37): python3-cryptography-2.3-3.el8.x86_64. 15 MB/s | 511 kB 00:00 2021-01-15T18:36:28.721 INFO:teuthology.orchestra.run.smithi148.stdout:(29/37): python3-webencodings-0.5.1-6.el8.noarc 1.1 MB/s | 27 kB 00:00 2021-01-15T18:36:28.753 INFO:teuthology.orchestra.run.smithi148.stdout:(30/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 42 MB/s | 1.5 MB 00:00 2021-01-15T18:36:28.754 INFO:teuthology.orchestra.run.smithi148.stdout:(31/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 3.1 MB/s | 103 kB 00:00 2021-01-15T18:36:28.779 INFO:teuthology.orchestra.run.smithi148.stdout:(32/37): python3-werkzeug-0.12.2-4.el8.noarch.r 17 MB/s | 457 kB 00:00 2021-01-15T18:36:28.781 INFO:teuthology.orchestra.run.smithi148.stdout:(33/37): python3-mako-1.0.6-13.el8.noarch.rpm 5.6 MB/s | 157 kB 00:00 2021-01-15T18:36:28.804 INFO:teuthology.orchestra.run.smithi148.stdout:(34/37): python3-html5lib-0.999999999-6.el8.noa 8.7 MB/s | 214 kB 00:00 2021-01-15T18:36:28.811 INFO:teuthology.orchestra.run.smithi148.stdout:(35/37): python3-pytz-2017.2-9.el8.noarch.rpm 1.8 MB/s | 54 kB 00:00 2021-01-15T18:36:28.824 INFO:teuthology.orchestra.run.smithi148.stdout:(36/37): python3-markupsafe-0.23-19.el8.x86_64. 1.9 MB/s | 39 kB 00:00 2021-01-15T18:36:29.043 INFO:teuthology.orchestra.run.smithi148.stdout:(37/37): ceph-osd-16.0.0-9031.g892c51dd.el8.x86 20 MB/s | 18 MB 00:00 2021-01-15T18:36:29.046 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:36:29.047 INFO:teuthology.orchestra.run.smithi148.stdout:Total 27 MB/s | 33 MB 00:01 2021-01-15T18:36:29.058 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:36:29.097 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:36:29.098 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:36:29.429 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:36:29.433 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:36:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:36:30.911 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-01-15T18:36:31.088 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-01-15T18:36:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 3/37 2021-01-15T18:36:31.529 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 4/37 2021-01-15T18:36:31.747 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 5/37 2021-01-15T18:36:31.928 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/37 2021-01-15T18:36:32.166 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 7/37 2021-01-15T18:36:32.470 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 8/37 2021-01-15T18:36:32.889 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 9/37 2021-01-15T18:36:33.078 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 10/37 2021-01-15T18:36:33.319 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 11/37 2021-01-15T18:36:33.635 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 12/37 2021-01-15T18:36:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 13/37 2021-01-15T18:36:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 14/37 2021-01-15T18:36:34.192 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 15/37 2021-01-15T18:36:34.435 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-01-15T18:36:34.699 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-01-15T18:36:34.896 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-01-15T18:36:35.079 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-01-15T18:36:35.236 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-01-15T18:36:35.423 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-01-15T18:36:35.634 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-01-15T18:36:35.837 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.5.1-1.el8.noarch 23/37 2021-01-15T18:36:35.997 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-01-15T18:36:36.293 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-01-15T18:36:36.586 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-01-15T18:36:36.923 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-01-15T18:36:37.186 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-01-15T18:36:37.434 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-01-15T18:36:37.778 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-01-15T18:36:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-01-15T18:36:38.810 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8. 32/37 2021-01-15T18:36:38.996 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 33/37 2021-01-15T18:36:40.543 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 33/37 2021-01-15T18:36:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 34/37 2021-01-15T18:36:41.182 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 34/37 2021-01-15T18:36:41.412 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 35/37 2021-01-15T18:36:41.638 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 35/37 2021-01-15T18:36:41.799 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 36/37 2021-01-15T18:36:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 36/37 2021-01-15T18:36:41.930 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 37/37 2021-01-15T18:36:42.280 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 37/37 2021-01-15T18:36:42.281 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 1/37 2021-01-15T18:36:42.281 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 2/37 2021-01-15T18:36:42.281 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 3/37 2021-01-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 4/37 2021-01-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 5/37 2021-01-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8. 6/37 2021-01-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-01-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-01-15T18:36:42.284 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.5.1-1.el8.noarch 9/37 2021-01-15T18:36:42.284 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-01-15T18:36:42.285 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-01-15T18:36:42.285 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-01-15T18:36:42.285 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-01-15T18:36:42.286 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-01-15T18:36:42.286 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-01-15T18:36:42.287 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-01-15T18:36:42.287 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-01-15T18:36:42.287 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-01-15T18:36:42.288 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-01-15T18:36:42.288 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-01-15T18:36:42.289 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-01-15T18:36:42.289 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-01-15T18:36:42.289 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-01-15T18:36:42.290 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-01-15T18:36:42.290 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-01-15T18:36:42.291 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 26/37 2021-01-15T18:36:42.291 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 27/37 2021-01-15T18:36:42.291 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 28/37 2021-01-15T18:36:42.292 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 29/37 2021-01-15T18:36:42.292 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 30/37 2021-01-15T18:36:42.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 31/37 2021-01-15T18:36:42.293 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 32/37 2021-01-15T18:36:42.294 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 33/37 2021-01-15T18:36:42.294 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 34/37 2021-01-15T18:36:42.294 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 35/37 2021-01-15T18:36:42.295 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 36/37 2021-01-15T18:36:43.348 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 37/37 2021-01-15T18:36:43.348 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:36:45.190 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:45.190 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:36:45.190 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:45.191 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:45.191 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:45.191 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:36:45.191 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:45.192 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:36:45.192 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-01-15T18:36:45.192 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-01-15T18:36:45.192 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-01-15T18:36:45.192 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-01-15T18:36:45.193 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.5.1-1.el8.noarch 2021-01-15T18:36:45.193 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-01-15T18:36:45.193 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-01-15T18:36:45.193 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-01-15T18:36:45.194 INFO:teuthology.orchestra.run.smithi148.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-01-15T18:36:45.194 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-6.2-6.el8.noarch 2021-01-15T18:36:45.194 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-01-15T18:36:45.194 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-01-15T18:36:45.194 INFO:teuthology.orchestra.run.smithi148.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-01-15T18:36:45.195 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.0.6-13.el8.noarch 2021-01-15T18:36:45.195 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-01-15T18:36:45.195 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-01-15T18:36:45.195 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-01-15T18:36:45.196 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-2.6-1.el8.noarch 2021-01-15T18:36:45.196 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-01-15T18:36:45.196 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.14-14.el8.noarch 2021-01-15T18:36:45.196 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2017.2-9.el8.noarch 2021-01-15T18:36:45.196 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-01-15T18:36:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-01-15T18:36:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-01-15T18:36:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-01-15T18:36:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-01-15T18:36:45.198 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-01-15T18:36:45.198 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-01-15T18:36:45.198 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-01-15T18:36:45.198 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-01-15T18:36:45.199 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-01-15T18:36:45.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:45.199 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:36:45.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2021-01-15T18:36:45.617 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:36:46.096 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-01-15T18:36:46.206 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-01-15T18:36:47.357 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:36:47.676 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:36:47.676 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:36:47.677 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:36:47.754 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install cephadm 2021-01-15T18:36:48.034 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:36:48.530 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-01-15T18:36:48.639 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-01-15T18:36:50.109 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:36:50.110 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:50.110 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:36:50.111 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:50.111 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:36:50.111 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 66 k 2021-01-15T18:36:50.111 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:50.111 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:36:50.112 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:50.112 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:36:50.112 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:50.112 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 66 k 2021-01-15T18:36:50.112 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 262 k 2021-01-15T18:36:50.113 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:36:50.257 INFO:teuthology.orchestra.run.smithi148.stdout:cephadm-16.0.0-9031.g892c51dd.el8.noarch.rpm 449 kB/s | 66 kB 00:00 2021-01-15T18:36:50.258 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:36:50.258 INFO:teuthology.orchestra.run.smithi148.stdout:Total 445 kB/s | 66 kB 00:00 2021-01-15T18:36:50.258 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:36:50.265 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:36:50.265 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:36:50.266 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:36:50.271 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:36:50.317 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:36:52.347 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T18:36:52.524 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T18:36:52.685 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T18:36:53.145 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T18:36:53.145 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:36:55.025 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:55.026 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:36:55.026 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:36:55.026 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:55.026 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:36:55.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache 2021-01-15T18:36:56.034 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:36:56.578 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-01-15T18:36:56.689 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 86 kB/s | 2.8 kB 00:00 2021-01-15T18:36:58.156 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:36:58.157 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:58.157 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2021-01-15T18:36:58.158 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:58.158 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:36:58.158 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 163 k 2021-01-15T18:36:58.158 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:58.159 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:36:58.159 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:36:58.159 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:36:58.159 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:36:58.159 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 163 k 2021-01-15T18:36:58.160 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 587 k 2021-01-15T18:36:58.160 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:36:58.326 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-16.0.0-9031.g892c51 971 kB/s | 163 kB 00:00 2021-01-15T18:36:58.327 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:36:58.327 INFO:teuthology.orchestra.run.smithi148.stdout:Total 963 kB/s | 163 kB 00:00 2021-01-15T18:36:58.328 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:36:58.341 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:36:58.341 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:36:58.390 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:36:58.394 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:36:58.510 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:36:58.713 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T18:36:59.060 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T18:36:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T18:36:59.486 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:01.335 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:01.335 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:01.336 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:37:01.337 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:01.337 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:01.426 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2021-01-15T18:37:01.714 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:02.220 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-01-15T18:37:02.331 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-01-15T18:37:03.529 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:37:03.851 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:03.852 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:37:03.853 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:03.933 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2021-01-15T18:37:04.220 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-01-15T18:37:04.821 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-01-15T18:37:06.299 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:06.300 INFO:teuthology.orchestra.run.smithi148.stdout:=================================================================================================== 2021-01-15T18:37:06.300 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:37:06.300 INFO:teuthology.orchestra.run.smithi148.stdout:=================================================================================================== 2021-01-15T18:37:06.301 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 2.7 M 2021-01-15T18:37:06.301 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:37:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 24 k 2021-01-15T18:37:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 8.8 k 2021-01-15T18:37:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-01-15T18:37:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-01-15T18:37:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-01-15T18:37:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:06.303 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:06.303 INFO:teuthology.orchestra.run.smithi148.stdout:=================================================================================================== 2021-01-15T18:37:06.303 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2021-01-15T18:37:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:06.304 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.0 M 2021-01-15T18:37:06.304 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 14 M 2021-01-15T18:37:06.304 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:06.422 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): ceph-prometheus-alerts-16.0.0-9031.g892c 74 kB/s | 8.8 kB 00:00 2021-01-15T18:37:06.449 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): ceph-grafana-dashboards-16.0.0-9031.g892 164 kB/s | 24 kB 00:00 2021-01-15T18:37:06.482 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 561 kB/s | 34 kB 00:00 2021-01-15T18:37:06.512 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-01-15T18:37:06.546 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.0 MB/s | 196 kB 00:00 2021-01-15T18:37:06.623 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): ceph-mgr-dashboard-16.0.0-9031.g892c51dd 8.5 MB/s | 2.7 MB 00:00 2021-01-15T18:37:06.626 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:06.626 INFO:teuthology.orchestra.run.smithi148.stdout:Total 9.3 MB/s | 3.0 MB 00:00 2021-01-15T18:37:06.627 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:06.635 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:06.636 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:06.779 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:06.783 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:07.066 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:07.230 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-01-15T18:37:07.437 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-01-15T18:37:07.630 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-01-15T18:37:07.801 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.n 4/6 2021-01-15T18:37:08.854 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8. 5/6 2021-01-15T18:37:09.150 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 6/6 2021-01-15T18:37:09.371 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 6/6 2021-01-15T18:37:09.372 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8. 1/6 2021-01-15T18:37:09.372 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 2/6 2021-01-15T18:37:09.372 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.n 3/6 2021-01-15T18:37:09.372 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-01-15T18:37:09.373 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-01-15T18:37:09.821 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-01-15T18:37:09.821 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:11.751 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:11.751 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:11.751 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:11.751 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:11.752 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:11.752 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-01-15T18:37:11.752 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-01-15T18:37:11.752 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.4.1-12.el8.noarch 2021-01-15T18:37:11.753 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:11.753 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:12.259 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2021-01-15T18:37:12.546 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:13.027 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-01-15T18:37:13.136 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 92 kB/s | 2.8 kB 00:00 2021-01-15T18:37:14.617 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:14.618 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================================================== 2021-01-15T18:37:14.619 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:37:14.619 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================================================== 2021-01-15T18:37:14.619 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:14.619 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 8.6 M 2021-01-15T18:37:14.619 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:37:14.620 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-01-15T18:37:14.620 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-01-15T18:37:14.620 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-01-15T18:37:14.620 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-01-15T18:37:14.621 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-01-15T18:37:14.621 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-01-15T18:37:14.621 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-01-15T18:37:14.621 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:14.621 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:14.622 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================================================== 2021-01-15T18:37:14.622 INFO:teuthology.orchestra.run.smithi148.stdout:Install 8 Packages 2021-01-15T18:37:14.622 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:14.623 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 37 M 2021-01-15T18:37:14.623 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 196 M 2021-01-15T18:37:14.623 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:14.666 INFO:teuthology.orchestra.run.smithi148.stdout:(1/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 14 MB/s | 639 kB 00:00 2021-01-15T18:37:14.668 INFO:teuthology.orchestra.run.smithi148.stdout:(2/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 3.5 MB/s | 169 kB 00:00 2021-01-15T18:37:14.701 INFO:teuthology.orchestra.run.smithi148.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 7.8 MB/s | 225 kB 00:00 2021-01-15T18:37:14.756 INFO:teuthology.orchestra.run.smithi148.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 68 MB/s | 3.7 MB 00:00 2021-01-15T18:37:14.884 INFO:teuthology.orchestra.run.smithi148.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 67 MB/s | 15 MB 00:00 2021-01-15T18:37:14.917 INFO:teuthology.orchestra.run.smithi148.stdout:(6/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2021-01-15T18:37:14.941 INFO:teuthology.orchestra.run.smithi148.stdout:(7/8): openblas-0.3.3-5.el8.x86_64.rpm 75 MB/s | 4.3 MB 00:00 2021-01-15T18:37:15.237 INFO:teuthology.orchestra.run.smithi148.stdout:(8/8): ceph-mgr-diskprediction-local-16.0.0-903 14 MB/s | 8.6 MB 00:00 2021-01-15T18:37:15.238 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:15.239 INFO:teuthology.orchestra.run.smithi148.stdout:Total 59 MB/s | 37 MB 00:00 2021-01-15T18:37:15.247 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:15.266 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:15.266 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:15.589 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:15.593 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:16.160 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:16.310 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-01-15T18:37:16.425 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-01-15T18:37:16.560 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-01-15T18:37:17.068 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-01-15T18:37:17.218 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-01-15T18:37:17.719 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-01-15T18:37:17.893 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.3-5.el8.x86_64 4/8 2021-01-15T18:37:18.637 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 4/8 2021-01-15T18:37:18.977 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-01-15T18:37:19.209 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-01-15T18:37:21.260 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-01-15T18:37:22.594 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-01-15T18:37:22.984 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 8/8 2021-01-15T18:37:23.249 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 8/8 2021-01-15T18:37:23.249 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 1/8 2021-01-15T18:37:23.249 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 2/8 2021-01-15T18:37:23.250 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 3/8 2021-01-15T18:37:23.250 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2021-01-15T18:37:23.251 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-01-15T18:37:23.251 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2021-01-15T18:37:23.252 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 7/8 2021-01-15T18:37:23.754 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 8/8 2021-01-15T18:37:23.755 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-01-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-01-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.3-5.el8.x86_64 2021-01-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-01-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-01-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-01-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-01-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:25.704 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2021-01-15T18:37:25.987 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:26.470 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-01-15T18:37:26.579 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-01-15T18:37:28.025 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:28.026 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================== 2021-01-15T18:37:28.026 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:37:28.027 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================== 2021-01-15T18:37:28.027 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:28.027 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 27 k 2021-01-15T18:37:28.027 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:37:28.028 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-01-15T18:37:28.028 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-01-15T18:37:28.028 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-01-15T18:37:28.028 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-01-15T18:37:28.028 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-01-15T18:37:28.029 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-01-15T18:37:28.029 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-01-15T18:37:28.029 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-01-15T18:37:28.029 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-01-15T18:37:28.029 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-01-15T18:37:28.030 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-01-15T18:37:28.030 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-01-15T18:37:28.030 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:28.030 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:28.030 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================================== 2021-01-15T18:37:28.031 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2021-01-15T18:37:28.031 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:28.031 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.9 M 2021-01-15T18:37:28.032 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2021-01-15T18:37:28.032 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:28.079 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 330 kB/s | 16 kB 00:00 2021-01-15T18:37:28.088 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 556 kB/s | 33 kB 00:00 2021-01-15T18:37:28.127 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.6 MB/s | 81 kB 00:00 2021-01-15T18:37:28.144 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 4.3 MB/s | 73 kB 00:00 2021-01-15T18:37:28.152 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): ceph-mgr-rook-16.0.0-9031.g892c51dd.el8 223 kB/s | 27 kB 00:00 2021-01-15T18:37:28.162 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 3.4 MB/s | 61 kB 00:00 2021-01-15T18:37:28.186 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2021-01-15T18:37:28.203 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.7 MB/s | 155 kB 00:00 2021-01-15T18:37:28.210 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 1.1 MB/s | 27 kB 00:00 2021-01-15T18:37:28.230 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 8.2 MB/s | 1.2 MB 00:00 2021-01-15T18:37:28.231 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 725 kB/s | 20 kB 00:00 2021-01-15T18:37:28.239 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.2 MB/s | 126 kB 00:00 2021-01-15T18:37:28.261 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 3.5 MB/s | 110 kB 00:00 2021-01-15T18:37:28.262 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:28.262 INFO:teuthology.orchestra.run.smithi148.stdout:Total 8.2 MB/s | 1.9 MB 00:00 2021-01-15T18:37:28.267 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:28.281 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:28.281 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:28.374 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:28.379 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:28.874 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:29.134 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-01-15T18:37:29.358 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-01-15T18:37:29.615 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-01-15T18:37:29.816 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2021-01-15T18:37:30.044 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2021-01-15T18:37:30.307 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-01-15T18:37:30.552 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-01-15T18:37:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-01-15T18:37:30.932 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-01-15T18:37:31.126 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-01-15T18:37:32.271 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-01-15T18:37:32.525 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-01-15T18:37:32.678 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 13/13 2021-01-15T18:37:32.906 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 13/13 2021-01-15T18:37:32.907 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 1/13 2021-01-15T18:37:32.907 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-01-15T18:37:32.907 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-01-15T18:37:32.908 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-01-15T18:37:32.908 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-01-15T18:37:32.909 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-01-15T18:37:32.909 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-01-15T18:37:32.909 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2021-01-15T18:37:32.910 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2021-01-15T18:37:32.910 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-01-15T18:37:32.911 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-01-15T18:37:32.911 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2021-01-15T18:37:33.439 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 13/13 2021-01-15T18:37:33.440 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:35.255 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:35.256 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:35.256 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:35.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-01-15T18:37:35.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-01-15T18:37:35.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-01-15T18:37:35.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-01-15T18:37:35.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-01-15T18:37:35.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-01-15T18:37:35.258 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-01-15T18:37:35.258 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-01-15T18:37:35.258 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-01-15T18:37:35.258 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-01-15T18:37:35.258 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-01-15T18:37:35.259 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-01-15T18:37:35.259 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:35.259 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:35.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-cephadm 2021-01-15T18:37:35.644 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:36.132 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-01-15T18:37:36.240 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-01-15T18:37:37.750 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:37.751 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2021-01-15T18:37:37.751 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:37:37.752 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2021-01-15T18:37:37.752 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:37.752 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 75 k 2021-01-15T18:37:37.752 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:37:37.752 INFO:teuthology.orchestra.run.smithi148.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-01-15T18:37:37.753 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-01-15T18:37:37.753 INFO:teuthology.orchestra.run.smithi148.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-01-15T18:37:37.753 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-01-15T18:37:37.753 INFO:teuthology.orchestra.run.smithi148.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-01-15T18:37:37.754 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:37.754 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:37.754 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2021-01-15T18:37:37.754 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2021-01-15T18:37:37.754 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:37.755 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 5.8 M 2021-01-15T18:37:37.755 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 24 M 2021-01-15T18:37:37.755 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:37.803 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 430 kB/s | 21 kB 00:00 2021-01-15T18:37:37.835 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.4 MB/s | 48 kB 00:00 2021-01-15T18:37:37.869 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 15 MB/s | 538 kB 00:00 2021-01-15T18:37:37.877 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.8 MB/s | 355 kB 00:00 2021-01-15T18:37:37.916 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-babel-2.5.1-5.el8.noarch.rpm 104 MB/s | 4.8 MB 00:00 2021-01-15T18:37:37.928 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): ceph-mgr-cephadm-16.0.0-9031.g892c51dd.e 428 kB/s | 75 kB 00:00 2021-01-15T18:37:37.930 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:37.930 INFO:teuthology.orchestra.run.smithi148.stdout:Total 33 MB/s | 5.8 MB 00:00 2021-01-15T18:37:37.933 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:37.941 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:37.941 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:38.017 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:38.021 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:39.070 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:39.422 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-01-15T18:37:39.713 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-01-15T18:37:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-01-15T18:37:40.220 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-01-15T18:37:40.483 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-01-15T18:37:40.656 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 6/6 2021-01-15T18:37:40.880 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 6/6 2021-01-15T18:37:40.881 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/6 2021-01-15T18:37:40.881 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-01-15T18:37:40.881 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-01-15T18:37:40.881 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-01-15T18:37:40.881 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-01-15T18:37:41.333 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 6/6 2021-01-15T18:37:41.333 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:43.258 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:43.259 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:43.259 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T18:37:43.259 INFO:teuthology.orchestra.run.smithi148.stdout: python3-apipkg-1.5-6.el8.noarch 2021-01-15T18:37:43.260 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.5.1-5.el8.noarch 2021-01-15T18:37:43.260 INFO:teuthology.orchestra.run.smithi148.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-01-15T18:37:43.260 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-01-15T18:37:43.260 INFO:teuthology.orchestra.run.smithi148.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-01-15T18:37:43.261 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:43.261 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:43.345 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2021-01-15T18:37:43.634 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:44.220 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-01-15T18:37:44.327 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-01-15T18:37:45.812 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:45.812 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================= 2021-01-15T18:37:45.813 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2021-01-15T18:37:45.813 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================= 2021-01-15T18:37:45.813 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:45.813 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 877 k 2021-01-15T18:37:45.813 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2021-01-15T18:37:45.814 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-01-15T18:37:45.814 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-01-15T18:37:45.814 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:45.814 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:45.814 INFO:teuthology.orchestra.run.smithi148.stdout:======================================================================================= 2021-01-15T18:37:45.815 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2021-01-15T18:37:45.815 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:45.815 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 981 k 2021-01-15T18:37:45.816 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 3.2 M 2021-01-15T18:37:45.816 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:45.853 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.1 MB/s | 83 kB 00:00 2021-01-15T18:37:45.855 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 517 kB/s | 21 kB 00:00 2021-01-15T18:37:46.407 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): ceph-fuse-16.0.0-9031.g892c51dd.el8.x86_ 1.4 MB/s | 877 kB 00:00 2021-01-15T18:37:46.407 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:46.408 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.6 MB/s | 981 kB 00:00 2021-01-15T18:37:46.410 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:46.431 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:46.432 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:46.507 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:46.511 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:46.704 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:46.704 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-01-15T18:37:46.704 INFO:teuthology.orchestra.run.smithi148.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-01-15T18:37:46.704 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:47.182 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-01-15T18:37:47.362 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 3/3 2021-01-15T18:37:47.684 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 3/3 2021-01-15T18:37:47.685 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/3 2021-01-15T18:37:47.685 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2021-01-15T18:37:48.163 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2021-01-15T18:37:48.164 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:50.017 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:50.017 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:50.017 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-01-15T18:37:50.017 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-01-15T18:37:50.018 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:50.018 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:50.129 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2021-01-15T18:37:50.421 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:50.946 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-01-15T18:37:51.065 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-01-15T18:37:52.546 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:52.546 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:37:52.547 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2021-01-15T18:37:52.547 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:37:52.547 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:37:52.547 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 131 k 2021-01-15T18:37:52.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:52.548 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:37:52.548 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:37:52.548 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:37:52.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:52.549 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 131 k 2021-01-15T18:37:52.549 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 437 k 2021-01-15T18:37:52.549 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:37:52.715 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-16.0.0-9031.g892c51dd.el8.x86_64 781 kB/s | 131 kB 00:00 2021-01-15T18:37:52.716 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:37:52.716 INFO:teuthology.orchestra.run.smithi148.stdout:Total 775 kB/s | 131 kB 00:00 2021-01-15T18:37:52.717 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:37:52.722 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:37:52.723 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:37:52.778 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:37:52.782 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:37:52.892 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:37:53.102 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:37:53.316 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:37:53.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:37:53.800 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:37:55.667 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:55.668 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:37:55.668 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:37:55.668 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:37:55.668 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:55.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2021-01-15T18:37:56.179 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:56.667 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-01-15T18:37:56.779 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2021-01-15T18:37:57.946 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:37:58.261 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:37:58.262 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:37:58.263 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:37:58.350 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2021-01-15T18:37:58.636 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:37:59.153 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-01-15T18:37:59.264 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-01-15T18:38:00.728 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:00.730 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:00.730 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2021-01-15T18:38:00.730 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:00.731 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:38:00.731 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 24 k 2021-01-15T18:38:00.731 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:00.731 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:38:00.732 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:00.732 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:38:00.732 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:00.733 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 24 k 2021-01-15T18:38:00.733 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 89 k 2021-01-15T18:38:00.733 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:38:01.089 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-16.0.0-9031.g892c51dd.el8.x86_6 66 kB/s | 24 kB 00:00 2021-01-15T18:38:01.089 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:38:01.090 INFO:teuthology.orchestra.run.smithi148.stdout:Total 66 kB/s | 24 kB 00:00 2021-01-15T18:38:01.090 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:38:01.094 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:38:01.094 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:38:01.096 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:38:01.101 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:38:01.156 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:38:01.335 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:01.540 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:02.022 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:02.023 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:38:03.858 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:03.858 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:38:03.859 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:38:03.859 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:03.859 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:03.919 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2021-01-15T18:38:04.207 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:04.718 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-01-15T18:38:04.828 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-01-15T18:38:06.003 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:06.324 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:06.325 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:06.326 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:06.405 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2021-01-15T18:38:06.690 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:07.186 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-01-15T18:38:07.299 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-01-15T18:38:08.453 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:08.774 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:08.775 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:08.775 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:08.829 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados 2021-01-15T18:38:09.117 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:09.599 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-01-15T18:38:09.706 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-01-15T18:38:10.877 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:11.192 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:11.193 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:11.193 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:11.244 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw 2021-01-15T18:38:11.531 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:12.053 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-01-15T18:38:12.164 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-01-15T18:38:13.310 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:13.625 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:13.626 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:13.626 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:13.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs 2021-01-15T18:38:13.961 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:14.462 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-01-15T18:38:14.572 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2021-01-15T18:38:15.752 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:16.066 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:16.067 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:16.068 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:16.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd 2021-01-15T18:38:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-01-15T18:38:17.006 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-01-15T18:38:18.143 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T18:38:18.457 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:18.458 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2021-01-15T18:38:18.458 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:18.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2021-01-15T18:38:18.797 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:19.275 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-01-15T18:38:19.381 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 kB/s | 2.8 kB 00:00 2021-01-15T18:38:20.832 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:20.833 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:20.834 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2021-01-15T18:38:20.834 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:20.834 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:38:20.834 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 89 k 2021-01-15T18:38:20.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:20.835 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:38:20.835 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:20.835 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:38:20.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:20.836 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 89 k 2021-01-15T18:38:20.836 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 298 k 2021-01-15T18:38:20.836 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:38:21.274 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-16.0.0-9031.g892c51dd.el8.x86_64.rpm 202 kB/s | 89 kB 00:00 2021-01-15T18:38:21.275 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:38:21.275 INFO:teuthology.orchestra.run.smithi148.stdout:Total 201 kB/s | 89 kB 00:00 2021-01-15T18:38:21.275 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:38:21.287 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:38:21.287 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:38:21.340 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:38:21.344 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:38:21.454 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:38:21.590 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:21.844 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:22.321 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:22.322 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:38:24.155 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:24.156 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:38:24.156 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:38:24.156 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:24.156 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:24.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2021-01-15T18:38:24.545 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:25.043 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-01-15T18:38:25.152 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-01-15T18:38:26.615 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:26.616 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:26.616 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2021-01-15T18:38:26.617 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:26.617 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:38:26.617 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.6 M 2021-01-15T18:38:26.617 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:26.618 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:38:26.618 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:26.618 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:38:26.618 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:26.619 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.6 M 2021-01-15T18:38:26.619 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 18 M 2021-01-15T18:38:26.619 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:38:26.950 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-16.0.0-9031.g892c51dd.el8.x86_64.rpm 11 MB/s | 3.6 MB 00:00 2021-01-15T18:38:26.950 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:38:26.951 INFO:teuthology.orchestra.run.smithi148.stdout:Total 11 MB/s | 3.6 MB 00:00 2021-01-15T18:38:26.951 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:38:26.964 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:38:26.964 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:38:27.032 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:38:27.036 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:38:27.541 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:38:27.732 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:28.093 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:28.566 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:28.567 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:38:30.389 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:30.390 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:38:30.390 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:38:30.390 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:30.391 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:30.468 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2021-01-15T18:38:30.761 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2021-01-15T18:38:31.290 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-01-15T18:38:31.424 INFO:teuthology.orchestra.run.smithi148.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-01-15T18:38:32.934 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2021-01-15T18:38:32.936 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:32.936 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2021-01-15T18:38:32.936 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:32.936 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2021-01-15T18:38:32.937 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 174 k 2021-01-15T18:38:32.937 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:32.937 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2021-01-15T18:38:32.937 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2021-01-15T18:38:32.938 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2021-01-15T18:38:32.938 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:32.938 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 174 k 2021-01-15T18:38:32.938 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 649 k 2021-01-15T18:38:32.938 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2021-01-15T18:38:33.521 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-16.0.0-9031.g892c51dd.el8.x86_64.rpm 297 kB/s | 174 kB 00:00 2021-01-15T18:38:33.521 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2021-01-15T18:38:33.522 INFO:teuthology.orchestra.run.smithi148.stdout:Total 296 kB/s | 174 kB 00:00 2021-01-15T18:38:33.522 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2021-01-15T18:38:33.535 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2021-01-15T18:38:33.535 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2021-01-15T18:38:33.591 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2021-01-15T18:38:33.595 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2021-01-15T18:38:33.740 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2021-01-15T18:38:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:34.153 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:34.634 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T18:38:34.635 INFO:teuthology.orchestra.run.smithi148.stdout:Installed products updated. 2021-01-15T18:38:36.511 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:36.512 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2021-01-15T18:38:36.512 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T18:38:36.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:36.512 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2021-01-15T18:38:36.613 DEBUG:teuthology.parallel:result is None 2021-01-15T18:38:36.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2021-01-15T18:38:36.648 INFO:teuthology.orchestra.run.smithi148.stdout:RedHatEnterprise 2021-01-15T18:38:36.649 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-01-15T18:38:36.649 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T18:38:36.834 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-01-15T18:38:36.855 INFO:teuthology.orchestra.run.smithi148.stdout:16.0.0-9031.g892c51dd.el8 2021-01-15T18:38:36.855 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-9031.g892c51dd.el8 2021-01-15T18:38:36.856 INFO:teuthology.task.install:The correct ceph version 16.0.0-9031.g892c51dd is installed. 2021-01-15T18:38:36.856 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-01-15T18:38:36.857 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:36.857 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-01-15T18:38:36.919 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-01-15T18:38:36.920 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:36.920 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2021-01-15T18:38:36.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-01-15T18:38:37.046 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-01-15T18:38:37.047 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:37.047 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2021-01-15T18:38:37.109 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-01-15T18:38:37.173 INFO:teuthology.run_tasks:Running task ceph... 2021-01-15T18:38:37.241 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-01-15T18:38:37.242 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2021-01-15T18:38:37.269 INFO:tasks.ceph:Disabling ceph logrotate... 2021-01-15T18:38:37.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2021-01-15T18:38:37.333 INFO:tasks.ceph:Creating extra log directories... 2021-01-15T18:38:37.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-01-15T18:38:37.428 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-01-15T18:38:37.429 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-01-15T18:38:37.429 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643', 'branch': 'master', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5788643', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'master', 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 80, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'master', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_master/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (1, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology_branch': 'master', 'timestamp': '2021-01-15_07:01:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.6159', 'targets': {'smithi148.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/Ata3o+j5f9cAKtufgWCn2Fw+o3874hUg/H93OmKwtXSpLfnN/Nj7Eu4k2CYhV4xhQra9OSwoZkC8sJdfD8YPvnsVAi+EUfuPwSibz10lD3sbFpBNMjmqo5srhnP+WGvWSPXmkCU0Zmo/qgPI573m6K0U0JXaha/Wy739fsnMhwmcMtTffyQLvjbGJX20f45058oj/SHV2E4uqaaieURLvR7TCLk9FqbymTpXE3uhEKvXeTmkkBariAyHNuL8xV7sIzlypniKI4TXqEUGz2dJ6Ib32634ByNEt7fbenySQRxh0AwJJnmUFH0krB7cK5NH7X/JZeEWr7c4+NYFr54DMCcPol1nlrFSXQYWoA2J2CaT6vJpt2J9YMFB3FOm7cj2wlpo0QXkpHh+x/lhEjjxfosve6fdxkbhrC4oubCzIdWyE/xQgCIQ1Njo4WK6c7Qq7vkAP5Uz2X4m3IlfWfFks5FNFlmnoh8sX+61YuSh1Mj06y9sCXofFDa75IzrCF8='}} 2021-01-15T18:38:37.430 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-01-15T18:38:37.451 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2021-01-15T18:38:37.515 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:37.516 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2021-01-15T18:38:37.569 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-01-15T18:38:37.570 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-01-15T18:38:37.623 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-01-15T18:38:37.624 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-01-15T18:38:37.624 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 105055 Links: 1 2021-01-15T18:38:37.624 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-01-15T18:38:37.624 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2021-01-15T18:38:37.625 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2021-01-15 18:38:27.912225365 +0000 2021-01-15T18:38:37.625 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2021-01-15 18:34:57.999829610 +0000 2021-01-15T18:38:37.625 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2021-01-15 18:34:57.999829610 +0000 2021-01-15T18:38:37.625 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2021-01-15T18:38:37.633 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2021-01-15T18:38:37.634 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2021-01-15T18:38:37.634 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000208438 s, 2.5 MB/s 2021-01-15T18:38:37.636 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-01-15T18:38:37.690 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-01-15T18:38:37.691 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-01-15T18:38:37.691 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 100884 Links: 1 2021-01-15T18:38:37.691 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-01-15T18:38:37.691 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2021-01-15T18:38:37.692 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2021-01-15 18:38:27.912225365 +0000 2021-01-15T18:38:37.692 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2021-01-15 18:34:58.315822361 +0000 2021-01-15T18:38:37.692 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2021-01-15 18:34:58.315822361 +0000 2021-01-15T18:38:37.692 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2021-01-15T18:38:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2021-01-15T18:38:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2021-01-15T18:38:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000240711 s, 2.1 MB/s 2021-01-15T18:38:37.702 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-01-15T18:38:37.757 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-01-15T18:38:37.757 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-01-15T18:38:37.757 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 105146 Links: 1 2021-01-15T18:38:37.757 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-01-15T18:38:37.758 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2021-01-15T18:38:37.758 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2021-01-15 18:38:27.912225365 +0000 2021-01-15T18:38:37.758 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2021-01-15 18:34:58.620815365 +0000 2021-01-15T18:38:37.758 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2021-01-15 18:34:58.620815365 +0000 2021-01-15T18:38:37.759 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2021-01-15T18:38:37.766 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2021-01-15T18:38:37.766 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2021-01-15T18:38:37.767 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.0002087 s, 2.5 MB/s 2021-01-15T18:38:37.769 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-01-15T18:38:37.822 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-01-15T18:38:37.823 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-01-15T18:38:37.823 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 105798 Links: 1 2021-01-15T18:38:37.823 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-01-15T18:38:37.824 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2021-01-15T18:38:37.824 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2021-01-15 18:38:27.912225365 +0000 2021-01-15T18:38:37.824 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2021-01-15 18:34:58.931808231 +0000 2021-01-15T18:38:37.824 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2021-01-15 18:34:58.931808231 +0000 2021-01-15T18:38:37.825 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2021-01-15T18:38:37.832 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2021-01-15T18:38:37.832 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2021-01-15T18:38:37.833 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000201589 s, 2.5 MB/s 2021-01-15T18:38:37.834 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-01-15T18:38:37.835 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2021-01-15T18:38:37.835 INFO:tasks.ceph:Generating config... 2021-01-15T18:38:37.836 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-01-15T18:38:37.837 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2021-01-15T18:38:37.837 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-01-15T18:38:37.837 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2021-01-15T18:38:37.838 INFO:tasks.ceph:[global] ms bind msgr1 = False 2021-01-15T18:38:37.838 INFO:tasks.ceph:[global] ms bind msgr2 = True 2021-01-15T18:38:37.839 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2021-01-15T18:38:37.839 INFO:tasks.ceph:[global] ms type = async 2021-01-15T18:38:37.839 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-01-15T18:38:37.840 INFO:tasks.ceph:[mgr] debug monc = 10 2021-01-15T18:38:37.841 INFO:tasks.ceph:[mgr] debug ms = 1 2021-01-15T18:38:37.842 INFO:tasks.ceph:[mon] debug mon = 20 2021-01-15T18:38:37.842 INFO:tasks.ceph:[mon] debug ms = 1 2021-01-15T18:38:37.842 INFO:tasks.ceph:[mon] debug paxos = 20 2021-01-15T18:38:37.843 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-01-15T18:38:37.843 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-01-15T18:38:37.843 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2021-01-15T18:38:37.843 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-01-15T18:38:37.844 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-01-15T18:38:37.844 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-01-15T18:38:37.844 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-01-15T18:38:37.845 INFO:tasks.ceph:[osd] debug filestore = 20 2021-01-15T18:38:37.845 INFO:tasks.ceph:[osd] debug journal = 20 2021-01-15T18:38:37.845 INFO:tasks.ceph:[osd] debug ms = 20 2021-01-15T18:38:37.845 INFO:tasks.ceph:[osd] debug osd = 25 2021-01-15T18:38:37.846 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-01-15T18:38:37.846 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-01-15T18:38:37.846 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-01-15T18:38:37.847 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-01-15T18:38:37.847 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-01-15T18:38:37.847 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-01-15T18:38:37.847 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-01-15T18:38:37.848 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-01-15T18:38:37.848 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-01-15T18:38:37.848 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-01-15T18:38:37.849 INFO:tasks.ceph:Setting up mon.a... 2021-01-15T18:38:37.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-01-15T18:38:37.913 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.keyring 2021-01-15T18:38:37.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-01-15T18:38:37.993 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-01-15T18:38:38.057 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.148'), ('mon.b', '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]'), ('mon.c', '[v2:172.21.15.148:3302,v1:172.21.15.148:6791]')] 2021-01-15T18:38:38.058 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790],[v2:172.21.15.148:3302,v1:172.21.15.148:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2021-01-15T18:38:38.059 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:38.059 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-01-15T18:38:38.111 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.148 --addv b '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]' --addv c '[v2:172.21.15.148:3302,v1:172.21.15.148:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:38.191 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:38.192 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid 327970d8-d19c-43f6-b60e-6ccadab35a2c 2021-01-15T18:38:38.192 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2021-01-15T18:38:38.192 INFO:teuthology.orchestra.run.smithi148.stdout:fsid 327970d8-d19c-43f6-b60e-6ccadab35a2c 2021-01-15T18:38:38.192 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2021-01-15T18:38:38.191821+0000 2021-01-15T18:38:38.193 INFO:teuthology.orchestra.run.smithi148.stdout:created 2021-01-15T18:38:38.191821+0000 2021-01-15T18:38:38.193 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 0 (unknown) 2021-01-15T18:38:38.193 INFO:teuthology.orchestra.run.smithi148.stdout:election_strategy: 3 2021-01-15T18:38:38.193 INFO:teuthology.orchestra.run.smithi148.stdout:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2021-01-15T18:38:38.193 INFO:teuthology.orchestra.run.smithi148.stdout:1: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.b 2021-01-15T18:38:38.194 INFO:teuthology.orchestra.run.smithi148.stdout:2: [v2:172.21.15.148:3302/0,v1:172.21.15.148:6791/0] mon.c 2021-01-15T18:38:38.194 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-01-15T18:38:38.195 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-01-15T18:38:38.248 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 327970d8-d19c-43f6-b60e-6ccadab35a2c... 2021-01-15T18:38:38.249 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-01-15T18:38:38.330 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2021-01-15T18:38:38.331 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2021-01-15T18:38:38.331 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-01-15T18:38:38.331 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2021-01-15T18:38:38.331 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.332 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2021-01-15T18:38:38.332 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.332 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2021-01-15T18:38:38.332 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.333 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2021-01-15T18:38:38.333 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2021-01-15T18:38:38.333 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.333 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2021-01-15T18:38:38.334 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2021-01-15T18:38:38.334 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.334 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2021-01-15T18:38:38.334 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2021-01-15T18:38:38.335 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.335 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2021-01-15T18:38:38.335 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2021-01-15T18:38:38.335 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.336 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2021-01-15T18:38:38.336 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.336 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2021-01-15T18:38:38.336 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2021-01-15T18:38:38.336 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2021-01-15T18:38:38.337 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2021-01-15T18:38:38.338 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2021-01-15T18:38:38.338 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2021-01-15T18:38:38.338 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2021-01-15T18:38:38.339 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-01-15T18:38:38.339 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2021-01-15T18:38:38.339 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2021-01-15T18:38:38.339 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.340 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-01-15T18:38:38.340 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.340 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2021-01-15T18:38:38.340 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.340 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2021-01-15T18:38:38.341 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.341 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2021-01-15T18:38:38.341 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2021-01-15T18:38:38.341 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2021-01-15T18:38:38.341 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790],[v2:172.21.15.148:3302,v1:172.21.15.148:6791]" 2021-01-15T18:38:38.342 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2021-01-15T18:38:38.342 INFO:teuthology.orchestra.run.smithi148.stdout: mon client hunt interval max multiple = 2 2021-01-15T18:38:38.342 INFO:teuthology.orchestra.run.smithi148.stdout: mon election default strategy = 3 2021-01-15T18:38:38.342 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr beacon grace = 90 2021-01-15T18:38:38.342 INFO:teuthology.orchestra.run.smithi148.stdout: ms bind msgr1 = False 2021-01-15T18:38:38.343 INFO:teuthology.orchestra.run.smithi148.stdout: ms bind msgr2 = True 2021-01-15T18:38:38.343 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 1000 2021-01-15T18:38:38.343 INFO:teuthology.orchestra.run.smithi148.stdout: ms type = async 2021-01-15T18:38:38.343 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 327970d8-d19c-43f6-b60e-6ccadab35a2c 2021-01-15T18:38:38.343 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.344 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2021-01-15T18:38:38.344 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2021-01-15T18:38:38.344 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.344 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2021-01-15T18:38:38.345 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2021-01-15T18:38:38.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.345 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2021-01-15T18:38:38.345 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2021-01-15T18:38:38.346 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.346 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2021-01-15T18:38:38.346 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2021-01-15T18:38:38.346 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2021-01-15T18:38:38.347 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.347 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2021-01-15T18:38:38.347 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2021-01-15T18:38:38.347 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.347 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2021-01-15T18:38:38.348 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.348 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2021-01-15T18:38:38.348 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.349 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2021-01-15T18:38:38.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.349 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2021-01-15T18:38:38.349 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2021-01-15T18:38:38.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.350 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2021-01-15T18:38:38.350 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2021-01-15T18:38:38.350 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2021-01-15T18:38:38.350 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression algorithm = zlib 2021-01-15T18:38:38.350 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression mode = aggressive 2021-01-15T18:38:38.351 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2021-01-15T18:38:38.351 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 20 2021-01-15T18:38:38.351 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 20 2021-01-15T18:38:38.351 INFO:teuthology.orchestra.run.smithi148.stdout: debug filestore = 20 2021-01-15T18:38:38.351 INFO:teuthology.orchestra.run.smithi148.stdout: debug journal = 20 2021-01-15T18:38:38.352 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 20 2021-01-15T18:38:38.352 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 25 2021-01-15T18:38:38.352 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 10 2021-01-15T18:38:38.352 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2021-01-15T18:38:38.352 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2021-01-15T18:38:38.353 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2021-01-15T18:38:38.353 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify cached snaps = True 2021-01-15T18:38:38.353 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify missing on start = True 2021-01-15T18:38:38.353 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2021-01-15T18:38:38.353 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2021-01-15T18:38:38.354 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue = debug_random 2021-01-15T18:38:38.354 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue cut off = debug_random 2021-01-15T18:38:38.354 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.354 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2021-01-15T18:38:38.355 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2021-01-15T18:38:38.355 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2021-01-15T18:38:38.355 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2021-01-15T18:38:38.355 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2021-01-15T18:38:38.355 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2021-01-15T18:38:38.356 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2021-01-15T18:38:38.356 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2021-01-15T18:38:38.356 INFO:teuthology.orchestra.run.smithi148.stdout: debug monc = 10 2021-01-15T18:38:38.356 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.356 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2021-01-15T18:38:38.357 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2021-01-15T18:38:38.357 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2021-01-15T18:38:38.357 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2021-01-15T18:38:38.357 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2021-01-15T18:38:38.357 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2021-01-15T18:38:38.358 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2021-01-15T18:38:38.358 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2021-01-15T18:38:38.358 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2021-01-15T18:38:38.358 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2021-01-15T18:38:38.358 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2021-01-15T18:38:38.359 INFO:teuthology.orchestra.run.smithi148.stdout: mon scrub interval = 300 2021-01-15T18:38:38.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:38:38.359 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2021-01-15T18:38:38.359 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2021-01-15T18:38:38.360 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2021-01-15T18:38:38.360 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2021-01-15T18:38:38.360 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-01-15T18:38:38.360 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-01-15T18:38:38.361 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2021-01-15T18:38:38.361 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2021-01-15T18:38:38.361 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2021-01-15T18:38:38.362 INFO:tasks.ceph:Creating admin key on mon.a... 2021-01-15T18:38:38.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-01-15T18:38:38.450 INFO:tasks.ceph:Copying monmap to all nodes... 2021-01-15T18:38:38.451 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:38.451 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-01-15T18:38:38.465 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:38.465 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-01-15T18:38:38.519 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:38:38.519 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:38.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.keyring 2021-01-15T18:38:38.520 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-01-15T18:38:38.593 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:38.593 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:38.647 INFO:tasks.ceph:Setting up mon nodes... 2021-01-15T18:38:38.648 INFO:tasks.ceph:Setting up mgr nodes... 2021-01-15T18:38:38.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-01-15T18:38:38.734 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-01-15T18:38:38.735 INFO:tasks.ceph:Setting up mds nodes... 2021-01-15T18:38:38.736 INFO:tasks.ceph_client:Setting up client nodes... 2021-01-15T18:38:38.736 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-01-15T18:38:38.737 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2021-01-15T18:38:38.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-01-15T18:38:38.760 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-01-15T18:38:38.761 INFO:tasks.ceph:role: osd.0 2021-01-15T18:38:38.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:38:38.761 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-01-15T18:38:38.832 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-01-15T18:38:38.832 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2021-01-15T18:38:38.832 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-01-15T18:38:38.832 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2021-01-15T18:38:38.833 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-01-15T18:38:38.833 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2021-01-15T18:38:38.833 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-01-15T18:38:38.833 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-01-15T18:38:38.834 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-01-15T18:38:38.834 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-01-15T18:38:39.195 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2021-01-15T18:38:39.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2021-01-15T18:38:39.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-01-15T18:38:39.314 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-01-15T18:38:39.343 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-01-15T18:38:39.407 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-01-15T18:38:39.407 INFO:tasks.ceph:role: osd.1 2021-01-15T18:38:39.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:38:39.408 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-01-15T18:38:39.477 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-01-15T18:38:39.478 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2021-01-15T18:38:39.478 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-01-15T18:38:39.478 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2021-01-15T18:38:39.479 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-01-15T18:38:39.479 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2021-01-15T18:38:39.479 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-01-15T18:38:39.479 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-01-15T18:38:39.479 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-01-15T18:38:39.480 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-01-15T18:38:39.843 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2021-01-15T18:38:39.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2021-01-15T18:38:39.845 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-01-15T18:38:39.931 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-01-15T18:38:39.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-01-15T18:38:40.023 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-01-15T18:38:40.024 INFO:tasks.ceph:role: osd.2 2021-01-15T18:38:40.024 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:38:40.025 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-01-15T18:38:40.093 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-01-15T18:38:40.094 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2021-01-15T18:38:40.094 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-01-15T18:38:40.094 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2021-01-15T18:38:40.095 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-01-15T18:38:40.095 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2021-01-15T18:38:40.095 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-01-15T18:38:40.095 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-01-15T18:38:40.095 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-01-15T18:38:40.096 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-01-15T18:38:40.460 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2021-01-15T18:38:40.462 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2021-01-15T18:38:40.462 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-01-15T18:38:40.561 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-01-15T18:38:40.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-01-15T18:38:40.652 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-01-15T18:38:40.652 INFO:tasks.ceph:role: osd.3 2021-01-15T18:38:40.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:38:40.653 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-01-15T18:38:40.724 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-01-15T18:38:40.725 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2021-01-15T18:38:40.725 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-01-15T18:38:40.725 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2021-01-15T18:38:40.726 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-01-15T18:38:40.726 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2021-01-15T18:38:40.727 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-01-15T18:38:40.727 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-01-15T18:38:40.727 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-01-15T18:38:40.728 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-01-15T18:38:41.093 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2021-01-15T18:38:41.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2021-01-15T18:38:41.095 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-01-15T18:38:41.193 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-01-15T18:38:41.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.300+0000 7f863e7e6080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-01-15T18:38:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.301+0000 7f863e7e6080 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-01-15T18:38:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.301+0000 7f863e7e6080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-01-15T18:38:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.301+0000 7f863e7e6080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-01-15T18:38:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.301+0000 7f863e7e6080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-01-15T18:38:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:41.309+0000 7f863e7e6080 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-01-15T18:38:44.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-01-15T18:38:44.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.315+0000 7f8028e9c080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-01-15T18:38:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.316+0000 7f8028e9c080 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-01-15T18:38:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.316+0000 7f8028e9c080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-01-15T18:38:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.316+0000 7f8028e9c080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-01-15T18:38:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.316+0000 7f8028e9c080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-01-15T18:38:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:44.316+0000 7f8028e9c080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-01-15T18:38:47.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-01-15T18:38:47.196 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:47.287 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.285+0000 7fe8ba135080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-01-15T18:38:47.287 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.285+0000 7fe8ba135080 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-01-15T18:38:47.288 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.285+0000 7fe8ba135080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-01-15T18:38:47.288 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.285+0000 7fe8ba135080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-01-15T18:38:47.288 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.285+0000 7fe8ba135080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-01-15T18:38:47.289 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:47.287+0000 7fe8ba135080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-01-15T18:38:50.133 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-01-15T18:38:50.162 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.246+0000 7f45ec681080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-01-15T18:38:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.246+0000 7f45ec681080 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-01-15T18:38:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.247+0000 7f45ec681080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-01-15T18:38:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.247+0000 7f45ec681080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-01-15T18:38:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.247+0000 7f45ec681080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-01-15T18:38:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:38:50.247+0000 7f45ec681080 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-01-15T18:38:53.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-01-15T18:38:53.122 INFO:tasks.ceph:Reading keys from all nodes... 2021-01-15T18:38:53.122 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:53.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-01-15T18:38:53.189 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:53.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-01-15T18:38:53.255 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:53.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-01-15T18:38:53.319 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:53.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-01-15T18:38:53.384 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:38:53.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-01-15T18:38:53.448 INFO:tasks.ceph:Adding keys to all mons... 2021-01-15T18:38:53.449 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2021-01-15T18:38:53.511 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2021-01-15T18:38:53.512 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAu4QFg7PrBKxAANuOxXLZ/Ycm8CM6n77uL0Q== 2021-01-15T18:38:53.512 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2021-01-15T18:38:53.512 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAx4QFge7gEEhAA6vUn0NODiQHFsJDKkdnlbQ== 2021-01-15T18:38:53.512 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2021-01-15T18:38:53.513 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQA04QFgzHXwEhAAIghk31X03lGzagKi2nN0xA== 2021-01-15T18:38:53.513 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2021-01-15T18:38:53.513 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQA34QFgMBQdERAAaIVbQ7c5pVLTpU00XjlfXQ== 2021-01-15T18:38:53.513 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2021-01-15T18:38:53.513 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQA64QFgKkPADhAAwQgI2GmlJ6bObNHjkznOVg== 2021-01-15T18:38:53.514 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-01-15T18:38:53.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-01-15T18:38:53.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-01-15T18:38:53.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-01-15T18:38:53.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-01-15T18:38:53.780 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-01-15T18:38:53.780 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-01-15T18:38:53.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-01-15T18:38:53.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-01-15T18:38:53.918 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-01-15T18:38:53.982 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-01-15T18:38:54.072 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-01-15T18:38:54.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-01-15T18:38:54.159 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-01-15T18:38:54.248 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-01-15T18:38:54.272 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-01-15T18:38:54.326 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-01-15T18:38:54.327 INFO:tasks.ceph.mon.a:Restarting daemon 2021-01-15T18:38:54.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-01-15T18:38:54.369 INFO:tasks.ceph.mon.a:Started 2021-01-15T18:38:54.369 INFO:tasks.ceph.mon.b:Restarting daemon 2021-01-15T18:38:54.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-01-15T18:38:54.372 INFO:tasks.ceph.mon.b:Started 2021-01-15T18:38:54.373 INFO:tasks.ceph.mon.c:Restarting daemon 2021-01-15T18:38:54.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-01-15T18:38:54.375 INFO:tasks.ceph.mon.c:Started 2021-01-15T18:38:54.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2021-01-15T18:39:06.584 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-01-15T18:39:06.585 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-01-15T18:39:06.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-01-15T18:39:06.588 INFO:tasks.ceph.mgr.x:Started 2021-01-15T18:39:06.588 INFO:tasks.ceph:Setting crush tunables to default 2021-01-15T18:39:06.589 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd crush tunables default 2021-01-15T18:39:06.792 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2021-01-15T18:39:06.802 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-01-15T18:39:06.803 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:39:06.803 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-01-15T18:39:06.827 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:39:06.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-01-15T18:39:06.891 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:39:06.892 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-01-15T18:39:06.955 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2021-01-15T18:39:06.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-01-15T18:39:07.019 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 107303da-44d7-4a60-915e-f6d1f55fa7cb 0 2021-01-15T18:39:07.255 INFO:teuthology.orchestra.run.smithi148.stdout:0 2021-01-15T18:39:07.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new d68efcd0-3df2-4b26-aa97-e09b936838a3 1 2021-01-15T18:39:07.464 INFO:teuthology.orchestra.run.smithi148.stdout:1 2021-01-15T18:39:07.477 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 755bf833-e4ce-4574-b5d3-90babe41a2b8 2 2021-01-15T18:39:07.677 INFO:teuthology.orchestra.run.smithi148.stdout:2 2021-01-15T18:39:07.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new c524c862-86a3-4d88-b92c-ecb219ba36f0 3 2021-01-15T18:39:07.933 INFO:teuthology.orchestra.run.smithi148.stdout:3 2021-01-15T18:39:07.946 INFO:tasks.ceph.osd.0:Restarting daemon 2021-01-15T18:39:07.946 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-01-15T18:39:07.949 INFO:tasks.ceph.osd.0:Started 2021-01-15T18:39:07.949 INFO:tasks.ceph.osd.1:Restarting daemon 2021-01-15T18:39:07.950 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-01-15T18:39:07.952 INFO:tasks.ceph.osd.1:Started 2021-01-15T18:39:07.952 INFO:tasks.ceph.osd.2:Restarting daemon 2021-01-15T18:39:07.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-01-15T18:39:07.955 INFO:tasks.ceph.osd.2:Started 2021-01-15T18:39:07.956 INFO:tasks.ceph.osd.3:Restarting daemon 2021-01-15T18:39:07.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-01-15T18:39:07.958 INFO:tasks.ceph.osd.3:Started 2021-01-15T18:39:07.959 INFO:tasks.ceph:Waiting for OSDs to come up 2021-01-15T18:39:08.362 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:08.539 INFO:teuthology.misc.health.smithi148.stdout: 2021-01-15T18:39:08.540 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":6,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:07.931524+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:08.550 DEBUG:teuthology.misc:0 of 4 OSDs are up 2021-01-15T18:39:08.551 INFO:tasks.ceph.osd.2.smithi148.stderr:2021-01-15T18:39:08.550+0000 7fbf4d2b7080 -1 Falling back to public interface 2021-01-15T18:39:08.556 INFO:tasks.ceph.osd.3.smithi148.stderr:2021-01-15T18:39:08.555+0000 7f8b06f61080 -1 Falling back to public interface 2021-01-15T18:39:08.564 INFO:tasks.ceph.osd.1.smithi148.stderr:2021-01-15T18:39:08.563+0000 7f6026412080 -1 Falling back to public interface 2021-01-15T18:39:08.572 INFO:tasks.ceph.osd.0.smithi148.stderr:2021-01-15T18:39:08.571+0000 7f9cb139b080 -1 Falling back to public interface 2021-01-15T18:39:09.883 INFO:tasks.ceph.osd.2.smithi148.stderr:2021-01-15T18:39:09.882+0000 7fbf4d2b7080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:39:09.887 INFO:tasks.ceph.osd.3.smithi148.stderr:2021-01-15T18:39:09.887+0000 7f8b06f61080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:39:09.897 INFO:tasks.ceph.osd.1.smithi148.stderr:2021-01-15T18:39:09.897+0000 7f6026412080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:39:09.907 INFO:tasks.ceph.osd.0.smithi148.stderr:2021-01-15T18:39:09.907+0000 7f9cb139b080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:39:12.098 INFO:tasks.ceph.osd.0.smithi148.stderr:2021-01-15T18:39:12.097+0000 7f9ca7b23700 -1 osd.0 0 waiting for initial osdmap 2021-01-15T18:39:12.098 INFO:tasks.ceph.osd.3.smithi148.stderr:2021-01-15T18:39:12.097+0000 7f8afd6e9700 -1 osd.3 0 waiting for initial osdmap 2021-01-15T18:39:12.100 INFO:tasks.ceph.osd.2.smithi148.stderr:2021-01-15T18:39:12.100+0000 7fbf43a3f700 -1 osd.2 0 waiting for initial osdmap 2021-01-15T18:39:12.101 INFO:tasks.ceph.osd.1.smithi148.stderr:2021-01-15T18:39:12.100+0000 7f601e39d700 -1 osd.1 0 waiting for initial osdmap 2021-01-15T18:39:14.954 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:15.218 INFO:teuthology.misc.health.smithi148.stdout: 2021-01-15T18:39:15.219 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:14.098211+0000","last_up_change":"2021-01-15T18:39:13.090654+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":31665}]},"public_addr":"172.21.15.148:6812/31665","cluster_addr":"172.21.15.148:6813/31665","heartbeat_back_addr":"172.21.15.148:6815/31665","heartbeat_front_addr":"172.21.15.148:6814/31665","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:15.230 DEBUG:teuthology.misc:4 of 4 OSDs are up 2021-01-15T18:39:15.230 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-01-15T18:39:15.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:15.233 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-01-15T18:39:15.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:15.497 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:14.098211+0000","last_up_change":"2021-01-15T18:39:13.090654+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":31665}]},"public_addr":"172.21.15.148:6812/31665","cluster_addr":"172.21.15.148:6813/31665","heartbeat_back_addr":"172.21.15.148:6815/31665","heartbeat_front_addr":"172.21.15.148:6814/31665","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:15.509 INFO:tasks.ceph.ceph_manager.ceph:[] 2021-01-15T18:39:15.510 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-01-15T18:39:15.510 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-01-15T18:39:15.511 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2021-01-15T18:39:15.803 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:15.815 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":4,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":31295}]},"active_addr":"172.21.15.148:6816/31295","active_change":"2021-01-15T18:39:10.345062+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":3789717297}]},{"addrvec":[{"type":"v2","addr":"172.21.15.148:0","nonce":2378837117}]}]}} 2021-01-15T18:39:15.819 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-01-15T18:39:15.819 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2021-01-15T18:39:15.820 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:16.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:16.077 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:14.098211+0000","last_up_change":"2021-01-15T18:39:13.090654+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":31665}]},"public_addr":"172.21.15.148:6812/31665","cluster_addr":"172.21.15.148:6813/31665","heartbeat_back_addr":"172.21.15.148:6815/31665","heartbeat_front_addr":"172.21.15.148:6814/31665","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:16.090 INFO:tasks.ceph.ceph_manager.ceph:all up! 2021-01-15T18:39:16.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:16.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:14.098211+0000","last_up_change":"2021-01-15T18:39:13.090654+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":31665}]},"public_addr":"172.21.15.148:6812/31665","cluster_addr":"172.21.15.148:6813/31665","heartbeat_back_addr":"172.21.15.148:6815/31665","heartbeat_front_addr":"172.21.15.148:6814/31665","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:16.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-01-15T18:39:16.505 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2021-01-15T18:39:16.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-01-15T18:39:16.646 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2021-01-15T18:39:16.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-01-15T18:39:16.787 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2021-01-15T18:39:16.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-01-15T18:39:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2021-01-15T18:39:16.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:17.193 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2021-01-15T18:39:17.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2021-01-15T18:39:18.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:18.481 INFO:teuthology.orchestra.run.smithi148.stdout:38654705666 2021-01-15T18:39:18.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2021-01-15T18:39:19.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:19.759 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2021-01-15T18:39:19.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2021-01-15T18:39:19.773 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-01-15T18:39:20.032 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2021-01-15T18:39:20.045 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705667 for osd.1 2021-01-15T18:39:20.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-01-15T18:39:20.306 INFO:teuthology.orchestra.run.smithi148.stdout:38654705667 2021-01-15T18:39:20.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705667 for osd.2 2021-01-15T18:39:20.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-01-15T18:39:20.583 INFO:teuthology.orchestra.run.smithi148.stdout:38654705668 2021-01-15T18:39:20.595 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705668 for osd.3 2021-01-15T18:39:20.595 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-01-15T18:39:20.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:21.712 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:21.714 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:21.727 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2021-01-15T18:39:20.350937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":19328,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":377468032,"statfs":{"total":386547056640,"available":386527264768,"internally_reserved":0,"allocated":360448,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]}]}]}} 2021-01-15T18:39:21.728 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:21.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:21.991 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:22.001 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2021-01-15T18:39:20.350937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":19328,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":377468032,"statfs":{"total":386547056640,"available":386527264768,"internally_reserved":0,"allocated":360448,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]}]}]}} 2021-01-15T18:39:22.002 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-01-15T18:39:22.002 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-01-15T18:39:22.002 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2021-01-15T18:39:22.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-01-15T18:39:22.312 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:22.324 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-01-15T18:39:22.325 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2021-01-15T18:39:22.325 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2021-01-15T18:39:22.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:22.598 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:22.599 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:14.098211+0000","last_up_change":"2021-01-15T18:39:13.090654+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":31665}]},"public_addr":"172.21.15.148:6812/31665","cluster_addr":"172.21.15.148:6813/31665","heartbeat_back_addr":"172.21.15.148:6815/31665","heartbeat_front_addr":"172.21.15.148:6814/31665","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:22.611 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2021-01-15T18:39:22.611 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2021-01-15T18:39:22.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:22.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:22.867 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:22.878 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-01-15T18:39:22.351171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":19328,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":377468032,"statfs":{"total":386547056640,"available":386527264768,"internally_reserved":0,"allocated":360448,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]}]}]}} 2021-01-15T18:39:22.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:23.132 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:23.145 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-01-15T18:39:22.351171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":19328,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":377468032,"statfs":{"total":386547056640,"available":386527264768,"internally_reserved":0,"allocated":360448,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4832,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94367008,"statfs":{"total":96636764160,"available":96631816192,"internally_reserved":0,"allocated":90112,"data_stored":18812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]}]}]}} 2021-01-15T18:39:23.146 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2021-01-15T18:39:23.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2021-01-15T18:39:23.491 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2021-01-15T18:39:23.491 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2021-01-15T18:39:25.508 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'unique_pool_0' created 2021-01-15T18:39:25.523 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2021-01-15T18:39:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:set pool 1 min_size to 2 2021-01-15T18:39:26.525 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2021-01-15T18:39:27.532 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rados' on pool 'unique_pool_0' 2021-01-15T18:39:28.549 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2021-01-15T18:39:28.550 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2021-01-15T18:39:28.602 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-01-15T18:39:28.748 INFO:teuthology.orchestra.run.smithi148.stdout:38654705670 2021-01-15T18:39:28.748 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-01-15T18:39:28.889 INFO:teuthology.orchestra.run.smithi148.stdout:38654705670 2021-01-15T18:39:28.890 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:29.150 INFO:teuthology.orchestra.run.smithi148.stdout:38654705669 2021-01-15T18:39:29.163 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705669 for osd.0 2021-01-15T18:39:30.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:30.429 INFO:teuthology.orchestra.run.smithi148.stdout:38654705669 2021-01-15T18:39:30.442 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705669 for osd.0 2021-01-15T18:39:31.444 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:31.715 INFO:teuthology.orchestra.run.smithi148.stdout:38654705671 2021-01-15T18:39:31.728 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705671 for osd.0 2021-01-15T18:39:31.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-01-15T18:39:31.987 INFO:teuthology.orchestra.run.smithi148.stdout:38654705670 2021-01-15T18:39:31.999 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705670 for osd.1 2021-01-15T18:39:32.000 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2021-01-15T18:39:32.000 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:32.256 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:32.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2021-01-15T18:39:30.352500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":20024,"kb_used_data":536,"kb_used_omap":0,"kb_used_meta":19456,"kb_avail":377467336,"statfs":{"total":386547056640,"available":386526552064,"internally_reserved":0,"allocated":548864,"data_stored":155790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19922944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529468+0000","last_change":"2021-01-15T18:39:27.529468+0000","last_active":"2021-01-15T18:39:27.529468+0000","last_peered":"2021-01-15T18:39:27.529468+0000","last_clean":"2021-01-15T18:39:27.529468+0000","last_became_active":"2021-01-15T18:39:27.528163+0000","last_became_peered":"2021-01-15T18:39:27.528163+0000","last_unstale":"2021-01-15T18:39:27.529468+0000","last_undegraded":"2021-01-15T18:39:27.529468+0000","last_fullsized":"2021-01-15T18:39:27.529468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.527079+0000","last_change":"2021-01-15T18:39:27.525872+0000","last_active":"2021-01-15T18:39:27.527079+0000","last_peered":"2021-01-15T18:39:27.527079+0000","last_clean":"2021-01-15T18:39:27.527079+0000","last_became_active":"2021-01-15T18:39:27.525330+0000","last_became_peered":"2021-01-15T18:39:27.525330+0000","last_unstale":"2021-01-15T18:39:27.527079+0000","last_undegraded":"2021-01-15T18:39:27.527079+0000","last_fullsized":"2021-01-15T18:39:27.527079+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516967+0000","last_change":"2021-01-15T18:39:27.516967+0000","last_active":"2021-01-15T18:39:27.516967+0000","last_peered":"2021-01-15T18:39:27.516967+0000","last_clean":"2021-01-15T18:39:27.516967+0000","last_became_active":"2021-01-15T18:39:27.516067+0000","last_became_peered":"2021-01-15T18:39:27.516067+0000","last_unstale":"2021-01-15T18:39:27.516967+0000","last_undegraded":"2021-01-15T18:39:27.516967+0000","last_fullsized":"2021-01-15T18:39:27.516967+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529889+0000","last_change":"2021-01-15T18:39:27.529889+0000","last_active":"2021-01-15T18:39:27.529889+0000","last_peered":"2021-01-15T18:39:27.529889+0000","last_clean":"2021-01-15T18:39:27.529889+0000","last_became_active":"2021-01-15T18:39:27.528536+0000","last_became_peered":"2021-01-15T18:39:27.528536+0000","last_unstale":"2021-01-15T18:39:27.529889+0000","last_undegraded":"2021-01-15T18:39:27.529889+0000","last_fullsized":"2021-01-15T18:39:27.529889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.530134+0000","last_change":"2021-01-15T18:39:27.527856+0000","last_active":"2021-01-15T18:39:27.530134+0000","last_peered":"2021-01-15T18:39:27.530134+0000","last_clean":"2021-01-15T18:39:27.530134+0000","last_became_active":"2021-01-15T18:39:27.527343+0000","last_became_peered":"2021-01-15T18:39:27.527343+0000","last_unstale":"2021-01-15T18:39:27.530134+0000","last_undegraded":"2021-01-15T18:39:27.530134+0000","last_fullsized":"2021-01-15T18:39:27.530134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526082+0000","last_change":"2021-01-15T18:39:27.524650+0000","last_active":"2021-01-15T18:39:27.526082+0000","last_peered":"2021-01-15T18:39:27.526082+0000","last_clean":"2021-01-15T18:39:27.526082+0000","last_became_active":"2021-01-15T18:39:27.523998+0000","last_became_peered":"2021-01-15T18:39:27.523998+0000","last_unstale":"2021-01-15T18:39:27.526082+0000","last_undegraded":"2021-01-15T18:39:27.526082+0000","last_fullsized":"2021-01-15T18:39:27.526082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526350+0000","last_change":"2021-01-15T18:39:27.525605+0000","last_active":"2021-01-15T18:39:27.526350+0000","last_peered":"2021-01-15T18:39:27.526350+0000","last_clean":"2021-01-15T18:39:27.526350+0000","last_became_active":"2021-01-15T18:39:27.524911+0000","last_became_peered":"2021-01-15T18:39:27.524911+0000","last_unstale":"2021-01-15T18:39:27.526350+0000","last_undegraded":"2021-01-15T18:39:27.526350+0000","last_fullsized":"2021-01-15T18:39:27.526350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.523659+0000","last_change":"2021-01-15T18:39:27.523109+0000","last_active":"2021-01-15T18:39:27.523659+0000","last_peered":"2021-01-15T18:39:27.523659+0000","last_clean":"2021-01-15T18:39:27.523659+0000","last_became_active":"2021-01-15T18:39:27.522455+0000","last_became_peered":"2021-01-15T18:39:27.522455+0000","last_unstale":"2021-01-15T18:39:27.523659+0000","last_undegraded":"2021-01-15T18:39:27.523659+0000","last_fullsized":"2021-01-15T18:39:27.523659+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526090+0000","last_change":"2021-01-15T18:39:27.524714+0000","last_active":"2021-01-15T18:39:27.526090+0000","last_peered":"2021-01-15T18:39:27.526090+0000","last_clean":"2021-01-15T18:39:27.526090+0000","last_became_active":"2021-01-15T18:39:27.524118+0000","last_became_peered":"2021-01-15T18:39:27.524118+0000","last_unstale":"2021-01-15T18:39:27.526090+0000","last_undegraded":"2021-01-15T18:39:27.526090+0000","last_fullsized":"2021-01-15T18:39:27.526090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529272+0000","last_change":"2021-01-15T18:39:27.529272+0000","last_active":"2021-01-15T18:39:27.529272+0000","last_peered":"2021-01-15T18:39:27.529272+0000","last_clean":"2021-01-15T18:39:27.529272+0000","last_became_active":"2021-01-15T18:39:27.528635+0000","last_became_peered":"2021-01-15T18:39:27.528635+0000","last_unstale":"2021-01-15T18:39:27.529272+0000","last_undegraded":"2021-01-15T18:39:27.529272+0000","last_fullsized":"2021-01-15T18:39:27.529272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.528455+0000","last_change":"2021-01-15T18:39:27.528455+0000","last_active":"2021-01-15T18:39:27.528455+0000","last_peered":"2021-01-15T18:39:27.528455+0000","last_clean":"2021-01-15T18:39:27.528455+0000","last_became_active":"2021-01-15T18:39:27.527623+0000","last_became_peered":"2021-01-15T18:39:27.527623+0000","last_unstale":"2021-01-15T18:39:27.528455+0000","last_undegraded":"2021-01-15T18:39:27.528455+0000","last_fullsized":"2021-01-15T18:39:27.528455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.523628+0000","last_change":"2021-01-15T18:39:27.522806+0000","last_active":"2021-01-15T18:39:27.523628+0000","last_peered":"2021-01-15T18:39:27.523628+0000","last_clean":"2021-01-15T18:39:27.523628+0000","last_became_active":"2021-01-15T18:39:27.522277+0000","last_became_peered":"2021-01-15T18:39:27.522277+0000","last_unstale":"2021-01-15T18:39:27.523628+0000","last_undegraded":"2021-01-15T18:39:27.523628+0000","last_fullsized":"2021-01-15T18:39:27.523628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516708+0000","last_change":"2021-01-15T18:39:27.513311+0000","last_active":"2021-01-15T18:39:27.516708+0000","last_peered":"2021-01-15T18:39:27.516708+0000","last_clean":"2021-01-15T18:39:27.516708+0000","last_became_active":"2021-01-15T18:39:27.512837+0000","last_became_peered":"2021-01-15T18:39:27.512837+0000","last_unstale":"2021-01-15T18:39:27.516708+0000","last_undegraded":"2021-01-15T18:39:27.516708+0000","last_fullsized":"2021-01-15T18:39:27.516708+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.527976+0000","last_change":"2021-01-15T18:39:27.526224+0000","last_active":"2021-01-15T18:39:27.527976+0000","last_peered":"2021-01-15T18:39:27.527976+0000","last_clean":"2021-01-15T18:39:27.527976+0000","last_became_active":"2021-01-15T18:39:27.525590+0000","last_became_peered":"2021-01-15T18:39:27.525590+0000","last_unstale":"2021-01-15T18:39:27.527976+0000","last_undegraded":"2021-01-15T18:39:27.527976+0000","last_fullsized":"2021-01-15T18:39:27.527976+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.530954+0000","last_change":"2021-01-15T18:39:27.530954+0000","last_active":"2021-01-15T18:39:27.530954+0000","last_peered":"2021-01-15T18:39:27.530954+0000","last_clean":"2021-01-15T18:39:27.530954+0000","last_became_active":"2021-01-15T18:39:27.530381+0000","last_became_peered":"2021-01-15T18:39:27.530381+0000","last_unstale":"2021-01-15T18:39:27.530954+0000","last_undegraded":"2021-01-15T18:39:27.530954+0000","last_fullsized":"2021-01-15T18:39:27.530954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516614+0000","last_change":"2021-01-15T18:39:27.516614+0000","last_active":"2021-01-15T18:39:27.516614+0000","last_peered":"2021-01-15T18:39:27.516614+0000","last_clean":"2021-01-15T18:39:27.516614+0000","last_became_active":"2021-01-15T18:39:27.515256+0000","last_became_peered":"2021-01-15T18:39:27.515256+0000","last_unstale":"2021-01-15T18:39:27.516614+0000","last_undegraded":"2021-01-15T18:39:27.516614+0000","last_fullsized":"2021-01-15T18:39:27.516614+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":35213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":35213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5012,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366828,"statfs":{"total":96636764160,"available":96631631872,"internally_reserved":0,"allocated":143360,"data_stored":42682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4850000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5012,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366828,"statfs":{"total":96636764160,"available":96631631872,"internally_reserved":0,"allocated":143360,"data_stored":42682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]}]}} 2021-01-15T18:39:32.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:32.526 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:32.528 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:32.540 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2021-01-15T18:39:32.352862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":20024,"kb_used_data":536,"kb_used_omap":0,"kb_used_meta":19456,"kb_avail":377467336,"statfs":{"total":386547056640,"available":386526552064,"internally_reserved":0,"allocated":548864,"data_stored":155790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19922944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529468+0000","last_change":"2021-01-15T18:39:27.529468+0000","last_active":"2021-01-15T18:39:27.529468+0000","last_peered":"2021-01-15T18:39:27.529468+0000","last_clean":"2021-01-15T18:39:27.529468+0000","last_became_active":"2021-01-15T18:39:27.528163+0000","last_became_peered":"2021-01-15T18:39:27.528163+0000","last_unstale":"2021-01-15T18:39:27.529468+0000","last_undegraded":"2021-01-15T18:39:27.529468+0000","last_fullsized":"2021-01-15T18:39:27.529468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.527079+0000","last_change":"2021-01-15T18:39:27.525872+0000","last_active":"2021-01-15T18:39:27.527079+0000","last_peered":"2021-01-15T18:39:27.527079+0000","last_clean":"2021-01-15T18:39:27.527079+0000","last_became_active":"2021-01-15T18:39:27.525330+0000","last_became_peered":"2021-01-15T18:39:27.525330+0000","last_unstale":"2021-01-15T18:39:27.527079+0000","last_undegraded":"2021-01-15T18:39:27.527079+0000","last_fullsized":"2021-01-15T18:39:27.527079+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516967+0000","last_change":"2021-01-15T18:39:27.516967+0000","last_active":"2021-01-15T18:39:27.516967+0000","last_peered":"2021-01-15T18:39:27.516967+0000","last_clean":"2021-01-15T18:39:27.516967+0000","last_became_active":"2021-01-15T18:39:27.516067+0000","last_became_peered":"2021-01-15T18:39:27.516067+0000","last_unstale":"2021-01-15T18:39:27.516967+0000","last_undegraded":"2021-01-15T18:39:27.516967+0000","last_fullsized":"2021-01-15T18:39:27.516967+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529889+0000","last_change":"2021-01-15T18:39:27.529889+0000","last_active":"2021-01-15T18:39:27.529889+0000","last_peered":"2021-01-15T18:39:27.529889+0000","last_clean":"2021-01-15T18:39:27.529889+0000","last_became_active":"2021-01-15T18:39:27.528536+0000","last_became_peered":"2021-01-15T18:39:27.528536+0000","last_unstale":"2021-01-15T18:39:27.529889+0000","last_undegraded":"2021-01-15T18:39:27.529889+0000","last_fullsized":"2021-01-15T18:39:27.529889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.530134+0000","last_change":"2021-01-15T18:39:27.527856+0000","last_active":"2021-01-15T18:39:27.530134+0000","last_peered":"2021-01-15T18:39:27.530134+0000","last_clean":"2021-01-15T18:39:27.530134+0000","last_became_active":"2021-01-15T18:39:27.527343+0000","last_became_peered":"2021-01-15T18:39:27.527343+0000","last_unstale":"2021-01-15T18:39:27.530134+0000","last_undegraded":"2021-01-15T18:39:27.530134+0000","last_fullsized":"2021-01-15T18:39:27.530134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526082+0000","last_change":"2021-01-15T18:39:27.524650+0000","last_active":"2021-01-15T18:39:27.526082+0000","last_peered":"2021-01-15T18:39:27.526082+0000","last_clean":"2021-01-15T18:39:27.526082+0000","last_became_active":"2021-01-15T18:39:27.523998+0000","last_became_peered":"2021-01-15T18:39:27.523998+0000","last_unstale":"2021-01-15T18:39:27.526082+0000","last_undegraded":"2021-01-15T18:39:27.526082+0000","last_fullsized":"2021-01-15T18:39:27.526082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526350+0000","last_change":"2021-01-15T18:39:27.525605+0000","last_active":"2021-01-15T18:39:27.526350+0000","last_peered":"2021-01-15T18:39:27.526350+0000","last_clean":"2021-01-15T18:39:27.526350+0000","last_became_active":"2021-01-15T18:39:27.524911+0000","last_became_peered":"2021-01-15T18:39:27.524911+0000","last_unstale":"2021-01-15T18:39:27.526350+0000","last_undegraded":"2021-01-15T18:39:27.526350+0000","last_fullsized":"2021-01-15T18:39:27.526350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.523659+0000","last_change":"2021-01-15T18:39:27.523109+0000","last_active":"2021-01-15T18:39:27.523659+0000","last_peered":"2021-01-15T18:39:27.523659+0000","last_clean":"2021-01-15T18:39:27.523659+0000","last_became_active":"2021-01-15T18:39:27.522455+0000","last_became_peered":"2021-01-15T18:39:27.522455+0000","last_unstale":"2021-01-15T18:39:27.523659+0000","last_undegraded":"2021-01-15T18:39:27.523659+0000","last_fullsized":"2021-01-15T18:39:27.523659+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.526090+0000","last_change":"2021-01-15T18:39:27.524714+0000","last_active":"2021-01-15T18:39:27.526090+0000","last_peered":"2021-01-15T18:39:27.526090+0000","last_clean":"2021-01-15T18:39:27.526090+0000","last_became_active":"2021-01-15T18:39:27.524118+0000","last_became_peered":"2021-01-15T18:39:27.524118+0000","last_unstale":"2021-01-15T18:39:27.526090+0000","last_undegraded":"2021-01-15T18:39:27.526090+0000","last_fullsized":"2021-01-15T18:39:27.526090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.529272+0000","last_change":"2021-01-15T18:39:27.529272+0000","last_active":"2021-01-15T18:39:27.529272+0000","last_peered":"2021-01-15T18:39:27.529272+0000","last_clean":"2021-01-15T18:39:27.529272+0000","last_became_active":"2021-01-15T18:39:27.528635+0000","last_became_peered":"2021-01-15T18:39:27.528635+0000","last_unstale":"2021-01-15T18:39:27.529272+0000","last_undegraded":"2021-01-15T18:39:27.529272+0000","last_fullsized":"2021-01-15T18:39:27.529272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.528455+0000","last_change":"2021-01-15T18:39:27.528455+0000","last_active":"2021-01-15T18:39:27.528455+0000","last_peered":"2021-01-15T18:39:27.528455+0000","last_clean":"2021-01-15T18:39:27.528455+0000","last_became_active":"2021-01-15T18:39:27.527623+0000","last_became_peered":"2021-01-15T18:39:27.527623+0000","last_unstale":"2021-01-15T18:39:27.528455+0000","last_undegraded":"2021-01-15T18:39:27.528455+0000","last_fullsized":"2021-01-15T18:39:27.528455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.523628+0000","last_change":"2021-01-15T18:39:27.522806+0000","last_active":"2021-01-15T18:39:27.523628+0000","last_peered":"2021-01-15T18:39:27.523628+0000","last_clean":"2021-01-15T18:39:27.523628+0000","last_became_active":"2021-01-15T18:39:27.522277+0000","last_became_peered":"2021-01-15T18:39:27.522277+0000","last_unstale":"2021-01-15T18:39:27.523628+0000","last_undegraded":"2021-01-15T18:39:27.523628+0000","last_fullsized":"2021-01-15T18:39:27.523628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516708+0000","last_change":"2021-01-15T18:39:27.513311+0000","last_active":"2021-01-15T18:39:27.516708+0000","last_peered":"2021-01-15T18:39:27.516708+0000","last_clean":"2021-01-15T18:39:27.516708+0000","last_became_active":"2021-01-15T18:39:27.512837+0000","last_became_peered":"2021-01-15T18:39:27.512837+0000","last_unstale":"2021-01-15T18:39:27.516708+0000","last_undegraded":"2021-01-15T18:39:27.516708+0000","last_fullsized":"2021-01-15T18:39:27.516708+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.527976+0000","last_change":"2021-01-15T18:39:27.526224+0000","last_active":"2021-01-15T18:39:27.527976+0000","last_peered":"2021-01-15T18:39:27.527976+0000","last_clean":"2021-01-15T18:39:27.527976+0000","last_became_active":"2021-01-15T18:39:27.525590+0000","last_became_peered":"2021-01-15T18:39:27.525590+0000","last_unstale":"2021-01-15T18:39:27.527976+0000","last_undegraded":"2021-01-15T18:39:27.527976+0000","last_fullsized":"2021-01-15T18:39:27.527976+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.530954+0000","last_change":"2021-01-15T18:39:27.530954+0000","last_active":"2021-01-15T18:39:27.530954+0000","last_peered":"2021-01-15T18:39:27.530954+0000","last_clean":"2021-01-15T18:39:27.530954+0000","last_became_active":"2021-01-15T18:39:27.530381+0000","last_became_peered":"2021-01-15T18:39:27.530381+0000","last_unstale":"2021-01-15T18:39:27.530954+0000","last_undegraded":"2021-01-15T18:39:27.530954+0000","last_fullsized":"2021-01-15T18:39:27.530954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"12","reported_epoch":"15","state":"active+clean","last_fresh":"2021-01-15T18:39:27.516614+0000","last_change":"2021-01-15T18:39:27.516614+0000","last_active":"2021-01-15T18:39:27.516614+0000","last_peered":"2021-01-15T18:39:27.516614+0000","last_clean":"2021-01-15T18:39:27.516614+0000","last_became_active":"2021-01-15T18:39:27.515256+0000","last_became_peered":"2021-01-15T18:39:27.515256+0000","last_unstale":"2021-01-15T18:39:27.516614+0000","last_undegraded":"2021-01-15T18:39:27.516614+0000","last_fullsized":"2021-01-15T18:39:27.516614+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":35213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":35213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5012,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366828,"statfs":{"total":96636764160,"available":96631631872,"internally_reserved":0,"allocated":143360,"data_stored":42682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4850000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5012,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366828,"statfs":{"total":96636764160,"available":96631631872,"internally_reserved":0,"allocated":143360,"data_stored":42682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]}]}} 2021-01-15T18:39:32.541 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2021-01-15T18:39:32.541 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2021-01-15T18:39:32.542 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2021-01-15T18:39:32.787 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2021-01-15T18:39:32.787 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2021-01-15T18:39:32.832 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2021-01-15T18:39:32.832 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2021-01-15T18:39:32.915 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2021-01-15T18:39:32.915 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2021-01-15T18:39:32.999 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2021-01-15T18:39:33.000 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2021-01-15T18:39:33.084 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2021-01-15T18:39:33.084 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2021-01-15T18:39:33.361 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:39:33.361+0000 7f93ed7ef700 0 --2- 172.21.15.148:0/1708417523 >> [v2:172.21.15.148:3302/0,v1:172.21.15.148:6791/0] conn(0x55aa744e55c0 0x55aa7457db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-01-15T18:39:33.368 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2021-01-15T18:39:33.368 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2021-01-15T18:39:33.413 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2021-01-15T18:39:33.414 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2021-01-15T18:39:33.458 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2021-01-15T18:39:33.459 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2021-01-15T18:39:33.502 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2021-01-15T18:39:33.503 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2021-01-15T18:39:33.586 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2021-01-15T18:39:33.587 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2021-01-15T18:39:33.671 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2021-01-15T18:39:33.672 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2021-01-15T18:39:33.754 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2021-01-15T18:39:33.754 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2021-01-15T18:39:33.838 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2021-01-15T18:39:33.839 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2021-01-15T18:39:33.882 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2021-01-15T18:39:33.883 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2021-01-15T18:39:34.167 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2021-01-15T18:39:34.167 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2021-01-15T18:39:34.211 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2021-01-15T18:39:34.211 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2021-01-15T18:39:34.297 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2021-01-15T18:39:34.297 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2021-01-15T18:39:34.340 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2021-01-15T18:39:34.340 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2021-01-15T18:39:34.423 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2021-01-15T18:39:34.423 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2021-01-15T18:39:34.507 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2021-01-15T18:39:34.507 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2021-01-15T18:39:34.591 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2021-01-15T18:39:34.591 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2021-01-15T18:39:34.676 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2021-01-15T18:39:34.676 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2021-01-15T18:39:34.760 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2021-01-15T18:39:34.761 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2021-01-15T18:39:34.843 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2021-01-15T18:39:34.843 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2021-01-15T18:39:34.926 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2021-01-15T18:39:34.927 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2021-01-15T18:39:35.010 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2021-01-15T18:39:35.010 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2021-01-15T18:39:35.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2021-01-15T18:39:35.224 INFO:teuthology.orchestra.run.smithi148.stdout:{} 2021-01-15T18:39:35.224 INFO:teuthology.orchestra.run.smithi148.stderr:osd_min_pg_log_entries = '100000000' (not observed, change may require restart) osd_recovery_delay_start = '1000.000000' 2021-01-15T18:39:35.235 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-01-15T18:39:35.235 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:39:35.264 INFO:tasks.ceph.osd.0:Stopped 2021-01-15T18:39:35.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2021-01-15T18:39:35.595 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:35.594+0000 7f3f1ecac700 -1 mon.c@2(leader).osd e17 definitely_dead 0 2021-01-15T18:39:35.598 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0 is already down. 2021-01-15T18:39:35.608 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-01-15T18:39:35.609 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:39:35.634 INFO:tasks.ceph.osd.3:Stopped 2021-01-15T18:39:35.635 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2021-01-15T18:39:35.943 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:35.942+0000 7f3f1ecac700 -1 mon.c@2(leader).osd e17 definitely_dead 0 2021-01-15T18:39:36.558 INFO:teuthology.orchestra.run.smithi148.stderr:marked down osd.3. 2021-01-15T18:39:36.572 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2021-01-15T18:39:36.573 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2021-01-15T18:39:37.569 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2021-01-15T18:39:37.570 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2021-01-15T18:39:37.617 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2021-01-15T18:39:37.617 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2021-01-15T18:39:37.661 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2021-01-15T18:39:37.661 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2021-01-15T18:39:37.704 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2021-01-15T18:39:37.704 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2021-01-15T18:39:37.787 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2021-01-15T18:39:37.787 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2021-01-15T18:39:37.869 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2021-01-15T18:39:37.869 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2021-01-15T18:39:37.953 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2021-01-15T18:39:37.953 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2021-01-15T18:39:38.037 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2021-01-15T18:39:38.037 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2021-01-15T18:39:38.119 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2021-01-15T18:39:38.120 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2021-01-15T18:39:38.203 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2021-01-15T18:39:38.203 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2021-01-15T18:39:38.285 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2021-01-15T18:39:38.285 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2021-01-15T18:39:38.369 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2021-01-15T18:39:38.369 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2021-01-15T18:39:38.453 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2021-01-15T18:39:38.453 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2021-01-15T18:39:38.535 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2021-01-15T18:39:38.536 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2021-01-15T18:39:38.621 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2021-01-15T18:39:38.622 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2021-01-15T18:39:38.666 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2021-01-15T18:39:38.666 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2021-01-15T18:39:38.748 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2021-01-15T18:39:38.749 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2021-01-15T18:39:38.831 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2021-01-15T18:39:38.832 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2021-01-15T18:39:38.915 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2021-01-15T18:39:38.917 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2021-01-15T18:39:39.199 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2021-01-15T18:39:39.199 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2021-01-15T18:39:39.242 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2021-01-15T18:39:39.243 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2021-01-15T18:39:39.325 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2021-01-15T18:39:39.325 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2021-01-15T18:39:39.407 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2021-01-15T18:39:39.407 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2021-01-15T18:39:39.491 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2021-01-15T18:39:39.491 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2021-01-15T18:39:39.573 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2021-01-15T18:39:39.573 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2021-01-15T18:39:39.656 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2021-01-15T18:39:39.656 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2021-01-15T18:39:39.740 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-01-15T18:39:39.740 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:39:39.810 INFO:tasks.ceph.osd.1:Stopped 2021-01-15T18:39:39.811 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2021-01-15T18:39:40.119 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:40.119+0000 7f3f1ecac700 -1 mon.c@2(leader).osd e19 definitely_dead 0 2021-01-15T18:39:40.632 INFO:teuthology.orchestra.run.smithi148.stderr:marked down osd.1. 2021-01-15T18:39:40.643 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2021-01-15T18:39:41.633 INFO:teuthology.orchestra.run.smithi148.stderr:marked osd lost in epoch 20 2021-01-15T18:39:41.646 INFO:tasks.ceph.osd.0:Restarting daemon 2021-01-15T18:39:41.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-01-15T18:39:41.649 INFO:tasks.ceph.osd.0:Started 2021-01-15T18:39:41.650 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2021-01-15T18:39:41.767 INFO:teuthology.orchestra.run.smithi148.stderr:no valid command found; 10 closest matches: 2021-01-15T18:39:41.767 INFO:teuthology.orchestra.run.smithi148.stderr:0 2021-01-15T18:39:41.768 INFO:teuthology.orchestra.run.smithi148.stderr:1 2021-01-15T18:39:41.768 INFO:teuthology.orchestra.run.smithi148.stderr:2 2021-01-15T18:39:41.768 INFO:teuthology.orchestra.run.smithi148.stderr:abort 2021-01-15T18:39:41.768 INFO:teuthology.orchestra.run.smithi148.stderr:assert 2021-01-15T18:39:41.768 INFO:teuthology.orchestra.run.smithi148.stderr:config diff 2021-01-15T18:39:41.769 INFO:teuthology.orchestra.run.smithi148.stderr:config diff get 2021-01-15T18:39:41.769 INFO:teuthology.orchestra.run.smithi148.stderr:config get 2021-01-15T18:39:41.769 INFO:teuthology.orchestra.run.smithi148.stderr:config help [] 2021-01-15T18:39:41.769 INFO:teuthology.orchestra.run.smithi148.stderr:config set ... 2021-01-15T18:39:41.769 INFO:teuthology.orchestra.run.smithi148.stderr:admin_socket: invalid command 2021-01-15T18:39:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-01-15T18:39:41.771 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-01-15T18:39:42.241 INFO:tasks.ceph.osd.0.smithi148.stderr:2021-01-15T18:39:42.240+0000 7fbc533ec080 -1 Falling back to public interface 2021-01-15T18:39:43.590 INFO:tasks.ceph.osd.0.smithi148.stderr:2021-01-15T18:39:43.589+0000 7fbc533ec080 -1 osd.0 16 log_to_monitors {default=true} 2021-01-15T18:39:46.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2021-01-15T18:39:46.900 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:39:46.901 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2021-01-15T18:39:46.901 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2021-01-15T18:39:46.901 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:39:46.910 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2021-01-15T18:39:46.911 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:47.169 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:47.170 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":23,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:44.646118+0000","last_up_change":"2021-01-15T18:39:43.633878+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2021-01-15T18:39:24.508766+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":22,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":38392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":38392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":38392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":38392}]},"public_addr":"172.21.15.148:6804/38392","cluster_addr":"172.21.15.148:6805/38392","heartbeat_back_addr":"172.21.15.148:6807/38392","heartbeat_front_addr":"172.21.15.148:6806/38392","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":31656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":31656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":31656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":31656}]},"public_addr":"172.21.15.148:6804/31656","cluster_addr":"172.21.15.148:6805/31656","heartbeat_back_addr":"172.21.15.148:6807/31656","heartbeat_front_addr":"172.21.15.148:6806/31656","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:47.182 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2021-01-15T18:39:47.182 INFO:tasks.ceph.osd.3:Restarting daemon 2021-01-15T18:39:47.183 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-01-15T18:39:47.185 INFO:tasks.ceph.osd.3:Started 2021-01-15T18:39:47.185 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-01-15T18:39:47.298 INFO:teuthology.orchestra.run.smithi148.stderr:no valid command found; 10 closest matches: 2021-01-15T18:39:47.298 INFO:teuthology.orchestra.run.smithi148.stderr:0 2021-01-15T18:39:47.298 INFO:teuthology.orchestra.run.smithi148.stderr:1 2021-01-15T18:39:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:2 2021-01-15T18:39:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:abort 2021-01-15T18:39:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:assert 2021-01-15T18:39:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:config diff 2021-01-15T18:39:47.299 INFO:teuthology.orchestra.run.smithi148.stderr:config diff get 2021-01-15T18:39:47.300 INFO:teuthology.orchestra.run.smithi148.stderr:config get 2021-01-15T18:39:47.300 INFO:teuthology.orchestra.run.smithi148.stderr:config help [] 2021-01-15T18:39:47.300 INFO:teuthology.orchestra.run.smithi148.stderr:config set ... 2021-01-15T18:39:47.300 INFO:teuthology.orchestra.run.smithi148.stderr:admin_socket: invalid command 2021-01-15T18:39:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-01-15T18:39:47.301 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2021-01-15T18:39:47.769 INFO:tasks.ceph.osd.3.smithi148.stderr:2021-01-15T18:39:47.768+0000 7f20b79a6080 -1 Falling back to public interface 2021-01-15T18:39:48.653 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:48.653+0000 7f3f214b1700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 6 pgs recovery_unfound (PG_DAMAGED) 2021-01-15T18:39:49.121 INFO:tasks.ceph.osd.3.smithi148.stderr:2021-01-15T18:39:49.120+0000 7f20b79a6080 -1 osd.3 17 log_to_monitors {default=true} 2021-01-15T18:39:52.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-01-15T18:39:52.431 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:39:52.432 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2021-01-15T18:39:52.432 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2021-01-15T18:39:52.432 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:39:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2021-01-15T18:39:52.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:39:52.703 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:52.704 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":25,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:39:50.712479+0000","last_up_change":"2021-01-15T18:39:49.708296+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2021-01-15T18:39:24.508766+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":24,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":38392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":38392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":38392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":38392}]},"public_addr":"172.21.15.148:6804/38392","cluster_addr":"172.21.15.148:6805/38392","heartbeat_back_addr":"172.21.15.148:6807/38392","heartbeat_front_addr":"172.21.15.148:6806/38392","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":31680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":31680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":31680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":31680}]},"public_addr":"172.21.15.148:6808/31680","cluster_addr":"172.21.15.148:6809/31680","heartbeat_back_addr":"172.21.15.148:6811/31680","heartbeat_front_addr":"172.21.15.148:6810/31680","state":["exists"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":24,"up_thru":24,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":39252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":39252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":39252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":39252}]},"public_addr":"172.21.15.148:6808/39252","cluster_addr":"172.21.15.148:6809/39252","heartbeat_back_addr":"172.21.15.148:6811/39252","heartbeat_front_addr":"172.21.15.148:6810/39252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:39:52.716 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2021-01-15T18:39:52.716 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-01-15T18:39:52.858 INFO:teuthology.orchestra.run.smithi148.stdout:94489280515 2021-01-15T18:39:52.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-01-15T18:39:52.999 INFO:teuthology.orchestra.run.smithi148.stdout:38654705674 2021-01-15T18:39:53.000 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-01-15T18:39:53.141 INFO:teuthology.orchestra.run.smithi148.stdout:103079215106 2021-01-15T18:39:53.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:53.399 INFO:teuthology.orchestra.run.smithi148.stdout:94489280514 2021-01-15T18:39:53.412 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280515 got 94489280514 for osd.0 2021-01-15T18:39:54.414 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:54.462 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:54.461+0000 7f3f214b1700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2021-01-15T18:39:54.681 INFO:teuthology.orchestra.run.smithi148.stdout:94489280514 2021-01-15T18:39:54.693 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280515 got 94489280514 for osd.0 2021-01-15T18:39:55.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:55.967 INFO:teuthology.orchestra.run.smithi148.stdout:94489280516 2021-01-15T18:39:55.979 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280515 got 94489280516 for osd.0 2021-01-15T18:39:55.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-01-15T18:39:56.239 INFO:teuthology.orchestra.run.smithi148.stdout:38654705675 2021-01-15T18:39:56.251 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705674 got 38654705675 for osd.2 2021-01-15T18:39:56.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-01-15T18:39:56.511 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2021-01-15T18:39:56.523 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215106 got 103079215107 for osd.3 2021-01-15T18:39:56.524 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2021-01-15T18:39:56.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:39:56.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:56.782 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:39:56.792 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2021-01-15T18:39:56.358162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":21860,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":20798,"kb_avail":377465500,"statfs":{"total":386547056640,"available":386524672000,"internally_reserved":0,"allocated":1052672,"data_stored":502936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1944,"internal_metadata":21297256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":55,"num_objects_misplaced":0,"num_objects_unfound":18,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002101"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":"26","reported_epoch":"25","state":"active+undersized","last_fresh":"2021-01-15T18:39:50.742170+0000","last_change":"2021-01-15T18:39:50.742032+0000","last_active":"2021-01-15T18:39:50.742170+0000","last_peered":"2021-01-15T18:39:50.742170+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.742032+0000","last_became_peered":"2021-01-15T18:39:50.742032+0000","last_unstale":"2021-01-15T18:39:50.742170+0000","last_undegraded":"2021-01-15T18:39:50.742170+0000","last_fullsized":"2021-01-15T18:39:50.716998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":"40","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.749643+0000","last_change":"2021-01-15T18:39:50.750152+0000","last_active":"2021-01-15T18:39:50.749643+0000","last_peered":"2021-01-15T18:39:50.749643+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:39:50.733030+0000","last_became_peered":"2021-01-15T18:39:50.733030+0000","last_unstale":"2021-01-15T18:39:50.749643+0000","last_undegraded":"2021-01-15T18:39:50.724613+0000","last_fullsized":"2021-01-15T18:39:50.724211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":"54","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.745462+0000","last_change":"2021-01-15T18:39:50.745913+0000","last_active":"2021-01-15T18:39:50.745462+0000","last_peered":"2021-01-15T18:39:50.745462+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:39:50.729885+0000","last_became_peered":"2021-01-15T18:39:50.729885+0000","last_unstale":"2021-01-15T18:39:50.745462+0000","last_undegraded":"2021-01-15T18:39:50.716824+0000","last_fullsized":"2021-01-15T18:39:50.716337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"26","reported_epoch":"25","state":"active+undersized","last_fresh":"2021-01-15T18:39:50.739853+0000","last_change":"2021-01-15T18:39:50.739712+0000","last_active":"2021-01-15T18:39:50.739853+0000","last_peered":"2021-01-15T18:39:50.739853+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.739712+0000","last_became_peered":"2021-01-15T18:39:50.739712+0000","last_unstale":"2021-01-15T18:39:50.739853+0000","last_undegraded":"2021-01-15T18:39:50.739853+0000","last_fullsized":"2021-01-15T18:39:50.717064+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":"41","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.742766+0000","last_change":"2021-01-15T18:39:50.743312+0000","last_active":"2021-01-15T18:39:50.742766+0000","last_peered":"2021-01-15T18:39:50.742766+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:39:50.728122+0000","last_became_peered":"2021-01-15T18:39:50.728122+0000","last_unstale":"2021-01-15T18:39:50.742766+0000","last_undegraded":"2021-01-15T18:39:50.722745+0000","last_fullsized":"2021-01-15T18:39:50.722291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":"54","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.748842+0000","last_change":"2021-01-15T18:39:50.749196+0000","last_active":"2021-01-15T18:39:50.748842+0000","last_peered":"2021-01-15T18:39:50.748842+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:39:50.730670+0000","last_became_peered":"2021-01-15T18:39:50.730670+0000","last_unstale":"2021-01-15T18:39:50.748842+0000","last_undegraded":"2021-01-15T18:39:50.716754+0000","last_fullsized":"2021-01-15T18:39:50.716267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":"29","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.747582+0000","last_change":"2021-01-15T18:39:50.747991+0000","last_active":"2021-01-15T18:39:50.747582+0000","last_peered":"2021-01-15T18:39:50.747582+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:39:50.737732+0000","last_became_peered":"2021-01-15T18:39:50.737732+0000","last_unstale":"2021-01-15T18:39:50.747582+0000","last_undegraded":"2021-01-15T18:39:50.718708+0000","last_fullsized":"2021-01-15T18:39:50.718303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":"51","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.754867+0000","last_change":"2021-01-15T18:39:50.755195+0000","last_active":"2021-01-15T18:39:50.754867+0000","last_peered":"2021-01-15T18:39:50.754867+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:39:50.737428+0000","last_became_peered":"2021-01-15T18:39:50.737428+0000","last_unstale":"2021-01-15T18:39:50.754867+0000","last_undegraded":"2021-01-15T18:39:50.718079+0000","last_fullsized":"2021-01-15T18:39:50.716235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":"17","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.756605+0000","last_change":"2021-01-15T18:39:50.756964+0000","last_active":"2021-01-15T18:39:50.756605+0000","last_peered":"2021-01-15T18:39:50.756605+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:39:50.742321+0000","last_became_peered":"2021-01-15T18:39:50.742321+0000","last_unstale":"2021-01-15T18:39:50.756605+0000","last_undegraded":"2021-01-15T18:39:50.718397+0000","last_fullsized":"2021-01-15T18:39:50.718163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":"31","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.744140+0000","last_change":"2021-01-15T18:39:50.744540+0000","last_active":"2021-01-15T18:39:50.744140+0000","last_peered":"2021-01-15T18:39:50.744140+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:39:50.730601+0000","last_became_peered":"2021-01-15T18:39:50.730601+0000","last_unstale":"2021-01-15T18:39:50.744140+0000","last_undegraded":"2021-01-15T18:39:50.717607+0000","last_fullsized":"2021-01-15T18:39:50.717272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":"30","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.750571+0000","last_change":"2021-01-15T18:39:50.750979+0000","last_active":"2021-01-15T18:39:50.750571+0000","last_peered":"2021-01-15T18:39:50.750571+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:39:50.738893+0000","last_became_peered":"2021-01-15T18:39:50.738893+0000","last_unstale":"2021-01-15T18:39:50.750571+0000","last_undegraded":"2021-01-15T18:39:50.716614+0000","last_fullsized":"2021-01-15T18:39:50.715961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":"40","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.752621+0000","last_change":"2021-01-15T18:39:50.753043+0000","last_active":"2021-01-15T18:39:50.752621+0000","last_peered":"2021-01-15T18:39:50.752621+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:39:50.735502+0000","last_became_peered":"2021-01-15T18:39:50.735502+0000","last_unstale":"2021-01-15T18:39:50.752621+0000","last_undegraded":"2021-01-15T18:39:50.727060+0000","last_fullsized":"2021-01-15T18:39:50.726618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":"42","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.755734+0000","last_change":"2021-01-15T18:39:50.756113+0000","last_active":"2021-01-15T18:39:50.755734+0000","last_peered":"2021-01-15T18:39:50.755734+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:39:50.735947+0000","last_became_peered":"2021-01-15T18:39:50.735947+0000","last_unstale":"2021-01-15T18:39:50.755734+0000","last_undegraded":"2021-01-15T18:39:50.725685+0000","last_fullsized":"2021-01-15T18:39:50.725245+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":"41","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.746546+0000","last_change":"2021-01-15T18:39:50.747096+0000","last_active":"2021-01-15T18:39:50.746546+0000","last_peered":"2021-01-15T18:39:50.746546+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:39:50.731549+0000","last_became_peered":"2021-01-15T18:39:50.731549+0000","last_unstale":"2021-01-15T18:39:50.746546+0000","last_undegraded":"2021-01-15T18:39:50.723705+0000","last_fullsized":"2021-01-15T18:39:50.723244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":"35","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.753476+0000","last_change":"2021-01-15T18:39:50.753992+0000","last_active":"2021-01-15T18:39:50.753476+0000","last_peered":"2021-01-15T18:39:50.753476+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:39:50.739488+0000","last_became_peered":"2021-01-15T18:39:50.739488+0000","last_unstale":"2021-01-15T18:39:50.753476+0000","last_undegraded":"2021-01-15T18:39:50.716988+0000","last_fullsized":"2021-01-15T18:39:50.716568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":"52","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:39:50.751809+0000","last_change":"2021-01-15T18:39:50.752163+0000","last_active":"2021-01-15T18:39:50.751809+0000","last_peered":"2021-01-15T18:39:50.751809+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:39:50.731209+0000","last_became_peered":"2021-01-15T18:39:50.731209+0000","last_unstale":"2021-01-15T18:39:50.751809+0000","last_undegraded":"2021-01-15T18:39:50.717301+0000","last_fullsized":"2021-01-15T18:39:50.716845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":262144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5440,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":5183,"kb_avail":94366400,"statfs":{"total":96636764160,"available":96631193600,"internally_reserved":0,"allocated":253952,"data_stored":109161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":5307444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5640,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366200,"statfs":{"total":96636764160,"available":96630988800,"internally_reserved":0,"allocated":327680,"data_stored":182889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5504,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":5247,"kb_avail":94366336,"statfs":{"total":96636764160,"available":96631128064,"internally_reserved":0,"allocated":253952,"data_stored":109161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":5372980},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5276,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366564,"statfs":{"total":96636764160,"available":96631361536,"internally_reserved":0,"allocated":217088,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2021-01-15T18:39:56.794 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:39:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:57.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":47,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":25,"num_osds":4,"num_up_osds":3,"osd_up_since":1610735989,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":22384640,"bytes_avail":386524672000,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2021-01-15T18:39:04.488790+0000","services":{}},"progress_events":{}} 2021-01-15T18:39:57.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '327970d8-d19c-43f6-b60e-6ccadab35a2c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 47, 'monmap': {'epoch': 1, 'min_mon_release_name': 'pacific', 'num_mons': 3}, 'osdmap': {'epoch': 25, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1610735989, 'num_in_osds': 4, 'osd_in_since': 1610735953, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 117012, 'bytes_used': 22384640, 'bytes_avail': 386524672000, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2021-01-15T18:39:04.488790+0000', 'services': {}}, 'progress_events': {}} 2021-01-15T18:39:57.117 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2021-01-15T18:39:57.117 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-01-15T18:39:57.257 INFO:teuthology.orchestra.run.smithi148.stdout:94489280517 2021-01-15T18:39:57.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-01-15T18:39:57.398 INFO:teuthology.orchestra.run.smithi148.stdout:38654705677 2021-01-15T18:39:57.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-01-15T18:39:57.541 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2021-01-15T18:39:57.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:57.803 INFO:teuthology.orchestra.run.smithi148.stdout:94489280516 2021-01-15T18:39:57.815 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280517 got 94489280516 for osd.0 2021-01-15T18:39:58.817 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:39:59.086 INFO:teuthology.orchestra.run.smithi148.stdout:94489280517 2021-01-15T18:39:59.099 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280517 got 94489280517 for osd.0 2021-01-15T18:39:59.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-01-15T18:39:59.355 INFO:teuthology.orchestra.run.smithi148.stdout:38654705678 2021-01-15T18:39:59.368 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705678 for osd.2 2021-01-15T18:39:59.368 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-01-15T18:39:59.628 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2021-01-15T18:39:59.640 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215108 got 103079215108 for osd.3 2021-01-15T18:39:59.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:39:59.948 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:39:59.948 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":50,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":25,"num_osds":4,"num_up_osds":3,"osd_up_since":1610735989,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":22384640,"bytes_avail":386524672000,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2021-01-15T18:39:04.488790+0000","services":{}},"progress_events":{}} 2021-01-15T18:39:59.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': '327970d8-d19c-43f6-b60e-6ccadab35a2c', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 50, 'monmap': {'epoch': 1, 'min_mon_release_name': 'pacific', 'num_mons': 3}, 'osdmap': {'epoch': 25, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1610735989, 'num_in_osds': 4, 'osd_in_since': 1610735953, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 117012, 'bytes_used': 22384640, 'bytes_avail': 386524672000, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2021-01-15T18:39:04.488790+0000', 'services': {}}, 'progress_events': {}} 2021-01-15T18:39:59.961 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2021-01-15T18:39:59.961 DEBUG:teuthology.orchestra.run.smithi148:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2021-01-15T18:39:59.999 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:39:59.999+0000 7f3f214b1700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 27/28 objects unfound (96.429%); 1 osds down; Possible data damage: 14 pgs recovery_unfound; Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded 2021-01-15T18:40:00.004 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stderr:INFO: op_size has been rounded to 8192 2021-01-15T18:40:00.004 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:hints = 1 2021-01-15T18:40:00.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2021-01-15T18:40:00.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Object prefix: benchmark_data_smithi148_40831 2021-01-15T18:40:00.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:40:00.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:40:01.004 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 1 20 1065 1045 8.16322 8.16406 0.00483343 0.0168359 2021-01-15T18:40:02.004 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 2 20 2157 2137 8.34644 8.53125 0.0135228 0.0186139 2021-01-15T18:40:03.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 3 20 2635 2615 6.80907 3.73438 0.0230307 0.0228672 2021-01-15T18:40:04.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 4 20 3572 3552 6.93674 7.32031 0.0191382 0.0224572 2021-01-15T18:40:05.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 5 20 4509 4489 7.01335 7.32031 0.0240699 0.0222066 2021-01-15T18:40:06.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 6 20 5400 5380 7.00452 6.96094 0.0163873 0.0222634 2021-01-15T18:40:07.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 7 20 6472 6452 7.20021 8.375 0.0219155 0.0216668 2021-01-15T18:40:08.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 8 20 7115 7095 6.92807 5.02344 0.0217916 0.0225237 2021-01-15T18:40:09.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 9 20 7887 7867 6.82832 6.03125 0.00333363 0.0226256 2021-01-15T18:40:09.964 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:10.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 10 20 8402 8382 6.54779 4.02344 0.00857704 0.0233304 2021-01-15T18:40:10.223 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:10.224 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:10.235 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2021-01-15T18:40:08.361543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36210964,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":4488,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4461,"ondisk_log_size":4461,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":115480,"kb_used_data":51572,"kb_used_omap":1,"kb_used_meta":63870,"kb_avail":377371880,"statfs":{"total":386547056640,"available":386428805120,"internally_reserved":0,"allocated":52809728,"data_stored":52259992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1944,"internal_metadata":65402984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36093952,"num_objects":4406,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4406,"num_write_kb":35248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"1.f","version":"25'233","reported_seq":"259","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:04.128361+0000","last_change":"2021-01-15T18:40:00.048487+0000","last_active":"2021-01-15T18:40:04.128361+0000","last_peered":"2021-01-15T18:40:04.128361+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.742032+0000","last_became_peered":"2021-01-15T18:39:50.742032+0000","last_unstale":"2021-01-15T18:40:04.128361+0000","last_undegraded":"2021-01-15T18:40:00.031609+0000","last_fullsized":"2021-01-15T18:39:50.716998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":233,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1908736,"num_objects":233,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":1864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":233}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"25'201","reported_seq":"237","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:03.563155+0000","last_change":"2021-01-15T18:39:50.750152+0000","last_active":"2021-01-15T18:40:03.563155+0000","last_peered":"2021-01-15T18:40:03.563155+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:39:50.733030+0000","last_became_peered":"2021-01-15T18:39:50.733030+0000","last_unstale":"2021-01-15T18:40:03.563155+0000","last_undegraded":"2021-01-15T18:39:50.724613+0000","last_fullsized":"2021-01-15T18:39:50.724211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":201,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1622182,"num_objects":199,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":203,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":201,"num_write_kb":1596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":197},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"25'487","reported_seq":"538","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:08.094487+0000","last_change":"2021-01-15T18:39:50.745913+0000","last_active":"2021-01-15T18:40:08.094487+0000","last_peered":"2021-01-15T18:40:08.094487+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:39:50.729885+0000","last_became_peered":"2021-01-15T18:39:50.729885+0000","last_unstale":"2021-01-15T18:40:08.094487+0000","last_undegraded":"2021-01-15T18:39:50.716824+0000","last_fullsized":"2021-01-15T18:39:50.716337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":487,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3969107,"num_objects":485,"num_object_clones":0,"num_object_copies":1940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":487,"num_write_kb":3882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":484},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"25'227","reported_seq":"253","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:04.064032+0000","last_change":"2021-01-15T18:40:00.064227+0000","last_active":"2021-01-15T18:40:04.064032+0000","last_peered":"2021-01-15T18:40:04.064032+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.739712+0000","last_became_peered":"2021-01-15T18:39:50.739712+0000","last_unstale":"2021-01-15T18:40:04.064032+0000","last_undegraded":"2021-01-15T18:40:00.019710+0000","last_fullsized":"2021-01-15T18:39:50.717064+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":227,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1859584,"num_objects":227,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":1816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"25'226","reported_seq":"262","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:03.543287+0000","last_change":"2021-01-15T18:39:50.743312+0000","last_active":"2021-01-15T18:40:03.543287+0000","last_peered":"2021-01-15T18:40:03.543287+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:39:50.728122+0000","last_became_peered":"2021-01-15T18:39:50.728122+0000","last_unstale":"2021-01-15T18:40:03.543287+0000","last_undegraded":"2021-01-15T18:39:50.722745+0000","last_fullsized":"2021-01-15T18:39:50.722291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":226,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1818790,"num_objects":223,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":1788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":221},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"25'419","reported_seq":"469","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:08.095882+0000","last_change":"2021-01-15T18:39:50.749196+0000","last_active":"2021-01-15T18:40:08.095882+0000","last_peered":"2021-01-15T18:40:08.095882+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:39:50.730670+0000","last_became_peered":"2021-01-15T18:39:50.730670+0000","last_unstale":"2021-01-15T18:40:08.095882+0000","last_undegraded":"2021-01-15T18:39:50.716754+0000","last_fullsized":"2021-01-15T18:39:50.716267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":419,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3416230,"num_objects":418,"num_object_clones":0,"num_object_copies":1672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":422,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":419,"num_write_kb":3343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":416},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"25'231","reported_seq":"256","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:04.098486+0000","last_change":"2021-01-15T18:39:50.747991+0000","last_active":"2021-01-15T18:40:04.098486+0000","last_peered":"2021-01-15T18:40:04.098486+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:39:50.737732+0000","last_became_peered":"2021-01-15T18:39:50.737732+0000","last_unstale":"2021-01-15T18:40:04.098486+0000","last_undegraded":"2021-01-15T18:39:50.718708+0000","last_fullsized":"2021-01-15T18:39:50.718303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":231,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1867942,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":233,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":231,"num_write_kb":1836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":227},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"25'451","reported_seq":"500","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:08.079053+0000","last_change":"2021-01-15T18:39:50.755195+0000","last_active":"2021-01-15T18:40:08.079053+0000","last_peered":"2021-01-15T18:40:08.079053+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:39:50.737428+0000","last_became_peered":"2021-01-15T18:39:50.737428+0000","last_unstale":"2021-01-15T18:40:08.079053+0000","last_undegraded":"2021-01-15T18:39:50.718079+0000","last_fullsized":"2021-01-15T18:39:50.716235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":451,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3690579,"num_objects":451,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":453,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":451,"num_write_kb":3605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":450},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"25'229","reported_seq":"245","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:04.108257+0000","last_change":"2021-01-15T18:39:50.756964+0000","last_active":"2021-01-15T18:40:04.108257+0000","last_peered":"2021-01-15T18:40:04.108257+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:39:50.742321+0000","last_became_peered":"2021-01-15T18:39:50.742321+0000","last_unstale":"2021-01-15T18:40:04.108257+0000","last_undegraded":"2021-01-15T18:39:50.718397+0000","last_fullsized":"2021-01-15T18:39:50.718163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":229,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1871955,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":229,"num_write_kb":1829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":228},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"25'226","reported_seq":"252","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:04.081790+0000","last_change":"2021-01-15T18:39:50.744540+0000","last_active":"2021-01-15T18:40:04.081790+0000","last_peered":"2021-01-15T18:40:04.081790+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:39:50.730601+0000","last_became_peered":"2021-01-15T18:39:50.730601+0000","last_unstale":"2021-01-15T18:40:04.081790+0000","last_undegraded":"2021-01-15T18:39:50.717607+0000","last_fullsized":"2021-01-15T18:39:50.717272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":226,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1822969,"num_objects":224,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":230,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":1788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":221},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"25'224","reported_seq":"251","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:04.097006+0000","last_change":"2021-01-15T18:39:50.750979+0000","last_active":"2021-01-15T18:40:04.097006+0000","last_peered":"2021-01-15T18:40:04.097006+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:39:50.738893+0000","last_became_peered":"2021-01-15T18:39:50.738893+0000","last_unstale":"2021-01-15T18:40:04.097006+0000","last_undegraded":"2021-01-15T18:39:50.716614+0000","last_fullsized":"2021-01-15T18:39:50.715961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":224,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1818790,"num_objects":223,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":1783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":221},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'192","reported_seq":"227","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:03.594284+0000","last_change":"2021-01-15T18:39:50.753043+0000","last_active":"2021-01-15T18:40:03.594284+0000","last_peered":"2021-01-15T18:40:03.594284+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:39:50.735502+0000","last_became_peered":"2021-01-15T18:39:50.735502+0000","last_unstale":"2021-01-15T18:40:03.594284+0000","last_undegraded":"2021-01-15T18:39:50.727060+0000","last_fullsized":"2021-01-15T18:39:50.726618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":192,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1548454,"num_objects":190,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":194,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"25'206","reported_seq":"242","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:03.594596+0000","last_change":"2021-01-15T18:39:50.756113+0000","last_active":"2021-01-15T18:40:03.594596+0000","last_peered":"2021-01-15T18:40:03.594596+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:39:50.735947+0000","last_became_peered":"2021-01-15T18:39:50.735947+0000","last_unstale":"2021-01-15T18:40:03.594596+0000","last_undegraded":"2021-01-15T18:39:50.725685+0000","last_fullsized":"2021-01-15T18:39:50.725245+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":206,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1646758,"num_objects":202,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":206,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":1620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":200},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"25'209","reported_seq":"245","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:03.571036+0000","last_change":"2021-01-15T18:39:50.747096+0000","last_active":"2021-01-15T18:40:03.571036+0000","last_peered":"2021-01-15T18:40:03.571036+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:39:50.731549+0000","last_became_peered":"2021-01-15T18:39:50.731549+0000","last_unstale":"2021-01-15T18:40:03.571036+0000","last_undegraded":"2021-01-15T18:39:50.723705+0000","last_fullsized":"2021-01-15T18:39:50.723244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":209,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1679526,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":210,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":1652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":204},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"25'258","reported_seq":"283","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:04.097669+0000","last_change":"2021-01-15T18:39:50.753992+0000","last_active":"2021-01-15T18:40:04.097669+0000","last_peered":"2021-01-15T18:40:04.097669+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:39:50.739488+0000","last_became_peered":"2021-01-15T18:39:50.739488+0000","last_unstale":"2021-01-15T18:40:04.097669+0000","last_undegraded":"2021-01-15T18:39:50.716988+0000","last_fullsized":"2021-01-15T18:39:50.716568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":258,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2056524,"num_objects":253,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":259,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":2027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":250},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"25'442","reported_seq":"492","reported_epoch":"25","state":"active+recovery_unfound+undersized+degraded","last_fresh":"2021-01-15T18:40:08.094120+0000","last_change":"2021-01-15T18:39:50.752163+0000","last_active":"2021-01-15T18:40:08.094120+0000","last_peered":"2021-01-15T18:40:08.094120+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:39:50.731209+0000","last_became_peered":"2021-01-15T18:39:50.731209+0000","last_unstale":"2021-01-15T18:40:08.094120+0000","last_undegraded":"2021-01-15T18:39:50.717301+0000","last_fullsized":"2021-01-15T18:39:50.716845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":442,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3612838,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":446,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":3530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":440},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":36210964,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":4488,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":35478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57733120,"data_stored":57733120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4461,"ondisk_log_size":4461,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215110,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24848,"kb_used_data":10824,"kb_used_omap":0,"kb_used_meta":14015,"kb_avail":94346992,"statfs":{"total":96636764160,"available":96611319808,"internally_reserved":0,"allocated":11083776,"data_stored":10938985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":14351412},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.367}]}]},{"osd":2,"up_from":9,"seq":38654705680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59656,"kb_used_data":29116,"kb_used_omap":0,"kb_used_meta":30528,"kb_avail":94312184,"statfs":{"total":96636764160,"available":96575676416,"internally_reserved":0,"allocated":29814784,"data_stored":29669993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31260672},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280519,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25700,"kb_used_data":11420,"kb_used_omap":0,"kb_used_meta":14271,"kb_avail":94346140,"statfs":{"total":96636764160,"available":96610447360,"internally_reserved":0,"allocated":11694080,"data_stored":11549289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":14613556},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5276,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366564,"statfs":{"total":96636764160,"available":96631361536,"internally_reserved":0,"allocated":217088,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2021-01-15T18:40:10.237 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2021-01-15T18:40:10.237 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:10.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2021-01-15T18:40:10.423 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:10.424 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:10.425 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:10.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:10.427 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:10.427 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_4", 2021-01-15T18:40:10.427 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:10.427 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1224592654, 2021-01-15T18:40:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:10.429 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:10.429 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'1", 2021-01-15T18:40:10.429 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:10.429 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:10.429 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:10.430 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:10.430 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:10.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:10.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:10.430 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:10.431 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_6", 2021-01-15T18:40:10.431 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:10.431 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:10.431 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3101443614, 2021-01-15T18:40:10.431 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:10.432 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:10.432 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:10.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:10.432 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:10.432 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'2", 2021-01-15T18:40:10.433 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:10.433 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:10.433 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:10.433 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:10.433 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:10.434 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:10.434 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:10.434 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:10.434 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:10.435 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:10.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:10.435 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:10.435 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:10.435 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:10.436 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:10.436 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:10.436 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:10.452 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:10.452 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2021-01-15T18:40:10.453 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2021-01-15T18:40:10.787 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:10.807 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:10.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2021-01-15T18:40:11.005 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 11 20 8958 8938 6.34739 4.34375 0.00981097 0.0245873 2021-01-15T18:40:11.095 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:11.096 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:11.096 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:11.096 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:11.096 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:11.097 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:11.097 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:11.097 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:11.097 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:11.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.098 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 1, 2021-01-15T18:40:11.098 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 1, 2021-01-15T18:40:11.098 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:11.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:11.099 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:11.099 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_7", 2021-01-15T18:40:11.099 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:11.099 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:11.099 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 2358579165, 2021-01-15T18:40:11.100 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:11.100 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:11.100 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:11.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.100 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:11.101 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:11.101 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:11.101 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:11.101 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:11.102 INFO:teuthology.orchestra.run.smithi148.stdout: "2(0)" 2021-01-15T18:40:11.102 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:11.102 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:11.102 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:11.103 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:11.103 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:11.103 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:11.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:11.104 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(1)", 2021-01-15T18:40:11.104 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:11.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:11.104 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:11.104 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:11.105 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:11.126 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2021-01-15T18:40:11.126 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2021-01-15T18:40:11.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2021-01-15T18:40:11.478 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 1 objects unfound and apparently lost marking 2021-01-15T18:40:11.507 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2021-01-15T18:40:11.508 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:11.508 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2021-01-15T18:40:11.648 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:11.649 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:11.649 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:11.649 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:11.649 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:11.650 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:11.650 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:11.650 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:11.650 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:11.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.651 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:11.651 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:11.651 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:11.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:11.652 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:11.652 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_3", 2021-01-15T18:40:11.652 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:11.652 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:11.652 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 943960171, 2021-01-15T18:40:11.653 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:11.653 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:11.653 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:11.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.653 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'5", 2021-01-15T18:40:11.654 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:11.654 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:11.654 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:11.654 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:11.654 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:11.655 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:11.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:11.655 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:11.655 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_1", 2021-01-15T18:40:11.656 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:11.656 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:11.656 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 2885884443, 2021-01-15T18:40:11.656 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:11.656 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:11.657 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:11.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:11.657 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:11.657 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'1", 2021-01-15T18:40:11.657 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:11.658 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:11.658 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:11.658 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:11.658 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:11.658 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:11.659 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:11.659 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:11.659 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:11.659 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:11.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:11.660 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:11.660 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:11.660 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:11.660 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:11.660 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:11.661 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:11.662 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:11.662 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2021-01-15T18:40:11.662 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2021-01-15T18:40:11.948 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:11.979 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:11.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2021-01-15T18:40:12.007 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 12 20 9676 9656 6.28497 5.60938 0.0328794 0.024834 2021-01-15T18:40:12.233 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:12.233 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:12.234 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:12.234 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.234 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:12.234 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:12.234 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.235 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:12.235 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.235 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:12.235 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:12.236 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:12.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.236 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:12.236 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_9", 2021-01-15T18:40:12.236 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.237 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:12.237 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1217469130, 2021-01-15T18:40:12.237 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.237 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'1", 2021-01-15T18:40:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:12.239 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:12.239 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:12.239 INFO:teuthology.orchestra.run.smithi148.stdout: "2(1)" 2021-01-15T18:40:12.239 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:12.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.240 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:12.240 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_9", 2021-01-15T18:40:12.240 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.240 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:12.241 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 170713514, 2021-01-15T18:40:12.241 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.241 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:12.241 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.242 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'2", 2021-01-15T18:40:12.242 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:12.242 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:12.242 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:12.242 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: "2(1)" 2021-01-15T18:40:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:12.244 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:12.244 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:12.244 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.244 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:12.244 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:12.245 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:12.245 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:12.245 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:12.245 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:12.273 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:12.274 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2021-01-15T18:40:12.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2021-01-15T18:40:12.661 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:12.688 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:12.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2021-01-15T18:40:12.957 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:12.958 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:12.958 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:12.958 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.958 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:12.959 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:12.959 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.959 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:12.959 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.960 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:12.960 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:12.960 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:12.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.960 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:12.961 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_5", 2021-01-15T18:40:12.961 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.961 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:12.961 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 2998549577, 2021-01-15T18:40:12.961 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.962 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:12.962 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.962 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:12.963 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'2", 2021-01-15T18:40:12.963 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:12.963 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:12.963 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:12.963 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:12.964 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:12.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.964 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:12.965 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_2", 2021-01-15T18:40:12.965 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:12.965 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:12.965 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 4032093161, 2021-01-15T18:40:12.966 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:12.966 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:12.966 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:12.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:12.967 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:12.967 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'1", 2021-01-15T18:40:12.967 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:12.967 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:12.967 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:12.968 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:12.968 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:12.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:12.968 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:12.968 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:12.969 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:12.969 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:12.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:12.969 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:12.969 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:12.970 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:12.970 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:12.970 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:12.970 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:12.991 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:12.992 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2021-01-15T18:40:12.992 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2021-01-15T18:40:13.007 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 13 20 10542 10522 6.3219 6.76562 0.0192861 0.0246924 2021-01-15T18:40:13.399 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:13.413 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:13.413 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2021-01-15T18:40:13.778 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:13.778 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:13.778 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:13.779 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:13.779 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:13.779 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:13.779 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:13.779 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:13.780 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:13.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:13.780 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 1, 2021-01-15T18:40:13.780 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 1, 2021-01-15T18:40:13.780 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:13.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:13.781 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:13.781 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_8", 2021-01-15T18:40:13.781 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:13.781 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:13.782 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 2730327960, 2021-01-15T18:40:13.782 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:13.782 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:13.782 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:13.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:13.783 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'1", 2021-01-15T18:40:13.783 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:13.784 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:13.784 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:13.784 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:13.784 INFO:teuthology.orchestra.run.smithi148.stdout: "2(1)" 2021-01-15T18:40:13.785 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:13.785 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:13.785 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:13.786 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:13.786 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:13.786 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:13.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:13.787 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:13.787 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:13.788 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:13.788 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:13.788 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:13.789 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:13.805 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:13.806 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2021-01-15T18:40:13.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2021-01-15T18:40:14.007 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 14 20 11414 11394 6.35683 6.8125 0.020735 0.0245563 2021-01-15T18:40:14.094 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 1 objects unfound and apparently lost marking 2021-01-15T18:40:14.109 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:14.110 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2021-01-15T18:40:14.425 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:14.426 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:14.426 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:14.427 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.427 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:14.427 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:14.427 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.428 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:14.428 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.428 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 1, 2021-01-15T18:40:14.428 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 1, 2021-01-15T18:40:14.429 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:14.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.429 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:14.429 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_7", 2021-01-15T18:40:14.430 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.430 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:14.430 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3761017527, 2021-01-15T18:40:14.430 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.430 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:14.431 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.431 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'1", 2021-01-15T18:40:14.431 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:14.432 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:14.432 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:14.432 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:14.432 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:14.432 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:14.433 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:14.433 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:14.433 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:14.433 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:14.434 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:14.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.434 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:14.434 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:14.435 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:14.435 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:14.435 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:14.436 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:14.437 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:14.437 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2021-01-15T18:40:14.438 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2021-01-15T18:40:14.525 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:40:14.525+0000 7f3f214b1700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 9 pgs recovery_unfound (PG_DAMAGED) 2021-01-15T18:40:14.624 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 1 objects unfound and apparently lost marking 2021-01-15T18:40:14.648 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:14.649 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2021-01-15T18:40:14.983 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:14.984 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:14.984 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:14.984 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.984 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:14.985 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:14.985 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.985 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:14.985 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.986 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 3, 2021-01-15T18:40:14.986 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 3, 2021-01-15T18:40:14.986 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:14.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.987 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:14.987 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_8", 2021-01-15T18:40:14.987 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.987 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:14.987 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1861424966, 2021-01-15T18:40:14.988 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.988 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:14.988 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.988 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'5", 2021-01-15T18:40:14.989 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:14.989 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:14.989 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:14.989 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:14.989 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:14.990 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:14.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.990 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:14.990 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_6", 2021-01-15T18:40:14.991 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.991 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:14.991 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3910131350, 2021-01-15T18:40:14.991 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.991 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:14.992 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.992 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:14.992 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:14.992 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:14.993 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:14.993 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:14.993 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:14.993 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:14.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.994 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:14.994 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_4", 2021-01-15T18:40:14.994 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:14.995 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:14.995 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 2787839990, 2021-01-15T18:40:14.995 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:14.995 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:14.995 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:14.996 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:14.997 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:14.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:14.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:14.997 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:14.997 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:14.998 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:14.998 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:14.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:14.998 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(2)", 2021-01-15T18:40:14.998 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:14.999 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:14.999 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:14.999 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:14.999 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:15.008 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 15 20 12104 12084 6.29227 5.39062 0.0158834 0.0248093 2021-01-15T18:40:15.038 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2021-01-15T18:40:15.038 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2021-01-15T18:40:15.039 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2021-01-15T18:40:15.356 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 3 objects unfound and apparently lost marking 2021-01-15T18:40:15.381 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:15.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2021-01-15T18:40:15.681 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:15.682 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:15.682 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:15.682 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:15.683 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:15.683 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:15.683 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:15.683 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:15.683 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:15.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:15.684 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:15.684 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:15.684 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:15.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:15.685 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:15.685 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_3", 2021-01-15T18:40:15.685 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:15.685 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:15.685 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 459768081, 2021-01-15T18:40:15.686 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:15.686 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:15.686 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:15.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:15.687 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:15.687 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'1", 2021-01-15T18:40:15.687 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:15.687 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:15.687 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:15.688 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:15.688 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:15.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:15.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:15.689 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:15.689 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_3", 2021-01-15T18:40:15.689 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:15.689 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:15.690 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 480221105, 2021-01-15T18:40:15.690 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:15.690 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:15.690 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:15.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:15.691 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'2", 2021-01-15T18:40:15.691 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:15.691 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:15.691 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:15.691 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:15.692 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:15.692 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:15.692 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:15.692 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:15.692 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:15.693 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:15.693 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:15.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:15.693 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(1)", 2021-01-15T18:40:15.694 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:15.694 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:15.694 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:15.694 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:15.694 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:15.717 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2021-01-15T18:40:15.717 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2021-01-15T18:40:15.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2021-01-15T18:40:16.008 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 16 20 12941 12921 6.30758 6.53906 0.0299551 0.0247473 2021-01-15T18:40:16.041 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:16.077 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:16.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2021-01-15T18:40:16.367 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:16.368 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:16.368 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:16.368 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.369 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:16.369 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:16.369 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.369 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:16.369 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.371 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:16.371 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_5", 2021-01-15T18:40:16.371 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.371 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:16.372 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 4248945696, 2021-01-15T18:40:16.372 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.372 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:16.372 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.372 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.373 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'3", 2021-01-15T18:40:16.373 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:16.373 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:16.373 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:16.373 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:16.374 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:16.374 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:16.374 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.374 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.375 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:16.375 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_5", 2021-01-15T18:40:16.375 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.375 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:16.375 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 4212789968, 2021-01-15T18:40:16.376 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.376 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:16.376 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.376 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.376 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:16.377 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:16.377 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:16.377 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:16.377 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:16.377 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:16.378 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:16.378 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:16.378 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:16.378 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:16.378 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:16.379 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:16.379 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.379 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:16.379 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:16.379 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:16.380 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:16.380 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:16.414 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:16.414 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2021-01-15T18:40:16.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2021-01-15T18:40:16.583 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:16.594 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:16.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2021-01-15T18:40:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.896 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:16.899 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:16.900 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_6", 2021-01-15T18:40:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1708673474, 2021-01-15T18:40:16.901 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'5", 2021-01-15T18:40:16.902 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:16.903 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.904 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_9", 2021-01-15T18:40:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 831402034, 2021-01-15T18:40:16.905 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:16.906 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'6", 2021-01-15T18:40:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:16.907 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: "2(3)" 2021-01-15T18:40:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:16.908 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:16.909 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(2)", 2021-01-15T18:40:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:16.910 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:16.921 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2021-01-15T18:40:16.922 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2021-01-15T18:40:16.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2021-01-15T18:40:17.008 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 17 20 13641 13621 6.25816 5.46875 0.0240951 0.0249416 2021-01-15T18:40:17.339 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:17.371 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:17.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2021-01-15T18:40:17.689 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:17.689 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:17.690 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:17.690 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:17.690 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:17.690 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:17.690 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:17.691 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:17.691 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:17.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:17.691 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:17.692 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:17.692 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:17.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:17.692 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:17.692 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_8", 2021-01-15T18:40:17.693 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:17.693 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:17.693 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1901787523, 2021-01-15T18:40:17.693 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:17.693 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:17.694 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:17.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:17.694 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'5", 2021-01-15T18:40:17.694 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'3", 2021-01-15T18:40:17.695 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:17.695 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:17.695 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:17.695 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:17.695 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:17.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:17.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:17.696 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:17.696 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_1", 2021-01-15T18:40:17.696 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:17.697 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:17.697 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3122379363, 2021-01-15T18:40:17.698 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:17.698 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:17.698 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:17.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:17.699 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'4", 2021-01-15T18:40:17.699 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:17.699 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:17.700 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:17.700 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:17.700 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:17.700 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:17.700 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:17.701 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:17.701 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:17.701 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:17.701 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:17.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:17.702 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(0)", 2021-01-15T18:40:17.702 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:17.702 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:17.702 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:17.702 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:17.703 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:17.734 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2021-01-15T18:40:17.734 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2021-01-15T18:40:17.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2021-01-15T18:40:18.008 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 18 20 14477 14457 6.2733 6.53125 0.0370939 0.0248845 2021-01-15T18:40:18.072 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:18.100 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:18.100 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2021-01-15T18:40:18.451 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:18.452 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:18.452 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:18.452 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:18.453 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:18.453 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:18.453 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:18.453 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:18.453 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:18.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.454 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 3, 2021-01-15T18:40:18.454 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 3, 2021-01-15T18:40:18.454 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:18.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:18.455 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:18.455 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_4", 2021-01-15T18:40:18.455 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:18.455 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:18.456 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 873891716, 2021-01-15T18:40:18.456 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:18.456 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:18.456 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:18.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.457 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'8", 2021-01-15T18:40:18.457 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:18.457 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:18.457 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:18.457 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:18.458 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:18.458 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:18.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existing_7", 2021-01-15T18:40:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3996155460, 2021-01-15T18:40:18.460 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:18.460 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:18.460 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:18.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.460 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'9", 2021-01-15T18:40:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "16'6", 2021-01-15T18:40:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0", 2021-01-15T18:40:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:18.462 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:18.462 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:18.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:18.462 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:18.463 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "existed_2", 2021-01-15T18:40:18.463 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:18.463 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:18.463 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 1767667508, 2021-01-15T18:40:18.463 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:18.464 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:18.464 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:18.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:18.464 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'7", 2021-01-15T18:40:18.464 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:18.465 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:18.465 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:18.465 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:18.465 INFO:teuthology.orchestra.run.smithi148.stdout: "2(2)" 2021-01-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:18.467 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:18.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:18.467 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(3)", 2021-01-15T18:40:18.467 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:18.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:18.473 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:18.473 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:18.473 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:18.481 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [4179~18446744073709547436], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2021-01-15T18:40:18.481 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2021-01-15T18:40:18.482 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2021-01-15T18:40:18.771 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 3 objects unfound and apparently lost marking 2021-01-15T18:40:18.790 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2021-01-15T18:40:18.790 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2021-01-15T18:40:19.008 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 19 20 15137 15117 6.2145 5.15625 0.0152776 0.0251244 2021-01-15T18:40:19.074 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:19.074 INFO:teuthology.orchestra.run.smithi148.stdout: "offset": { 2021-01-15T18:40:19.075 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "", 2021-01-15T18:40:19.075 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:19.075 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": 0, 2021-01-15T18:40:19.075 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 0, 2021-01-15T18:40:19.076 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:19.076 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": -9223372036854775808, 2021-01-15T18:40:19.076 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:19.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:19.076 INFO:teuthology.orchestra.run.smithi148.stdout: "num_missing": 2, 2021-01-15T18:40:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: "num_unfound": 2, 2021-01-15T18:40:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: "objects": [ 2021-01-15T18:40:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_1", 2021-01-15T18:40:19.078 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:19.078 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:19.078 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 26281365, 2021-01-15T18:40:19.078 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:19.079 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:19.079 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:19.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:19.079 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'1", 2021-01-15T18:40:19.079 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:19.080 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:19.080 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:19.080 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:19.080 INFO:teuthology.orchestra.run.smithi148.stdout: "2(0)" 2021-01-15T18:40:19.080 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:19.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:19.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:19.081 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": { 2021-01-15T18:40:19.081 INFO:teuthology.orchestra.run.smithi148.stdout: "oid": "new_2", 2021-01-15T18:40:19.081 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2021-01-15T18:40:19.082 INFO:teuthology.orchestra.run.smithi148.stdout: "snapid": -2, 2021-01-15T18:40:19.082 INFO:teuthology.orchestra.run.smithi148.stdout: "hash": 3196340981, 2021-01-15T18:40:19.082 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 0, 2021-01-15T18:40:19.082 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2021-01-15T18:40:19.082 INFO:teuthology.orchestra.run.smithi148.stdout: "namespace": "" 2021-01-15T18:40:19.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2021-01-15T18:40:19.083 INFO:teuthology.orchestra.run.smithi148.stdout: "need": "19'2", 2021-01-15T18:40:19.083 INFO:teuthology.orchestra.run.smithi148.stdout: "have": "0'0", 2021-01-15T18:40:19.083 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "none", 2021-01-15T18:40:19.083 INFO:teuthology.orchestra.run.smithi148.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2021-01-15T18:40:19.084 INFO:teuthology.orchestra.run.smithi148.stdout: "locations": [ 2021-01-15T18:40:19.084 INFO:teuthology.orchestra.run.smithi148.stdout: "2(0)" 2021-01-15T18:40:19.084 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2021-01-15T18:40:19.084 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:19.084 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:19.085 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "NotRecovering", 2021-01-15T18:40:19.085 INFO:teuthology.orchestra.run.smithi148.stdout: "available_might_have_unfound": true, 2021-01-15T18:40:19.085 INFO:teuthology.orchestra.run.smithi148.stdout: "might_have_unfound": [ 2021-01-15T18:40:19.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2021-01-15T18:40:19.085 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": "1(3)", 2021-01-15T18:40:19.086 INFO:teuthology.orchestra.run.smithi148.stdout: "status": "osd is down" 2021-01-15T18:40:19.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2021-01-15T18:40:19.086 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2021-01-15T18:40:19.086 INFO:teuthology.orchestra.run.smithi148.stdout: "more": false 2021-01-15T18:40:19.087 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:19.103 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2021-01-15T18:40:19.103 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2021-01-15T18:40:19.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2021-01-15T18:40:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:pg has 2 objects unfound and apparently lost marking 2021-01-15T18:40:19.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2021-01-15T18:40:19.809 INFO:teuthology.orchestra.run.smithi148.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2021-01-15T18:40:19.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2021-01-15T18:40:20.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:40:20.009069+0000 min lat: 0.00287162 max lat: 0.624706 avg lat: 0.0251413 2021-01-15T18:40:20.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:40:20.010 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 20 20 15918 15898 6.20883 6.10156 0.0201665 0.0251413 2021-01-15T18:40:20.141 INFO:teuthology.orchestra.run.smithi148.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2021-01-15T18:40:20.164 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2021-01-15T18:40:20.334 INFO:teuthology.orchestra.run.smithi148.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2021-01-15T18:40:20.345 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-01-15T18:40:20.488 INFO:teuthology.orchestra.run.smithi148.stdout:94489280523 2021-01-15T18:40:20.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-01-15T18:40:20.560 INFO:tasks.ceph.mon.c.smithi148.stderr:2021-01-15T18:40:20.560+0000 7f3f214b1700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2021-01-15T18:40:20.638 INFO:teuthology.orchestra.run.smithi148.stdout:38654705683 2021-01-15T18:40:20.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-01-15T18:40:20.834 INFO:teuthology.orchestra.run.smithi148.stdout:103079215114 2021-01-15T18:40:20.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:40:21.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 21 20 16238 16218 6.03222 2.5 0.0052354 0.0257598 2021-01-15T18:40:21.368 INFO:teuthology.orchestra.run.smithi148.stdout:94489280522 2021-01-15T18:40:21.400 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280523 got 94489280522 for osd.0 2021-01-15T18:40:22.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 22 20 17046 17026 6.04487 6.3125 0.016572 0.0258286 2021-01-15T18:40:22.401 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-01-15T18:40:22.988 INFO:teuthology.orchestra.run.smithi148.stdout:94489280523 2021-01-15T18:40:23.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 23 20 17865 17845 6.06022 6.39844 0.0269703 0.0257631 2021-01-15T18:40:23.015 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 94489280523 got 94489280523 for osd.0 2021-01-15T18:40:23.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-01-15T18:40:23.374 INFO:teuthology.orchestra.run.smithi148.stdout:38654705683 2021-01-15T18:40:23.403 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705683 got 38654705683 for osd.2 2021-01-15T18:40:23.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-01-15T18:40:23.892 INFO:teuthology.orchestra.run.smithi148.stdout:103079215114 2021-01-15T18:40:23.924 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 103079215114 got 103079215114 for osd.3 2021-01-15T18:40:23.924 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2021-01-15T18:40:23.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:24.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 24 20 18347 18327 5.96461 3.76562 0.0293192 0.0261747 2021-01-15T18:40:24.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:24.322 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:24.339 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2021-01-15T18:40:22.365571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131762452,"num_objects":16098,"num_object_clones":0,"num_object_copies":64392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16125,"num_write_kb":128790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16152,"ondisk_log_size":16152,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":332652,"kb_used_data":192652,"kb_used_omap":1,"kb_used_meta":139966,"kb_avail":377154708,"statfs":{"total":386547056640,"available":386206420992,"internally_reserved":0,"allocated":197275648,"data_stored":196725912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1944,"internal_metadata":143325288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,5,12,10,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":69246976,"num_objects":8453,"num_object_clones":0,"num_object_copies":33812,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":8399,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":8453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":67624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"1.f","version":"25'967","reported_seq":"993","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.154525+0000","last_change":"2021-01-15T18:40:00.048487+0000","last_active":"2021-01-15T18:40:20.154525+0000","last_peered":"2021-01-15T18:40:20.154525+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.742032+0000","last_became_peered":"2021-01-15T18:39:50.742032+0000","last_unstale":"2021-01-15T18:40:20.154525+0000","last_undegraded":"2021-01-15T18:40:00.031609+0000","last_fullsized":"2021-01-15T18:39:50.716998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":967,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7921664,"num_objects":967,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":967,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":967,"num_write_kb":7736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"25'1014","reported_seq":"1060","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.177022+0000","last_change":"2021-01-15T18:40:10.885681+0000","last_active":"2021-01-15T18:40:20.177022+0000","last_peered":"2021-01-15T18:40:20.177022+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:39:50.733030+0000","last_became_peered":"2021-01-15T18:39:50.733030+0000","last_unstale":"2021-01-15T18:40:20.177022+0000","last_undegraded":"2021-01-15T18:39:50.724613+0000","last_fullsized":"2021-01-15T18:39:50.724211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1014,"ondisk_log_size":1014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8265894,"num_objects":1010,"num_object_clones":0,"num_object_copies":4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":8084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1010}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"25'1019","reported_seq":"1077","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.111078+0000","last_change":"2021-01-15T18:40:11.724276+0000","last_active":"2021-01-15T18:40:20.111078+0000","last_peered":"2021-01-15T18:40:20.111078+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:39:50.729885+0000","last_became_peered":"2021-01-15T18:39:50.729885+0000","last_unstale":"2021-01-15T18:40:20.111078+0000","last_undegraded":"2021-01-15T18:39:50.716824+0000","last_fullsized":"2021-01-15T18:39:50.716337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1019,"ondisk_log_size":1019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8319059,"num_objects":1016,"num_object_clones":0,"num_object_copies":4064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1018,"num_write_kb":8130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1016}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"25'1040","reported_seq":"1066","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.186803+0000","last_change":"2021-01-15T18:40:00.064227+0000","last_active":"2021-01-15T18:40:20.186803+0000","last_peered":"2021-01-15T18:40:20.186803+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.739712+0000","last_became_peered":"2021-01-15T18:39:50.739712+0000","last_unstale":"2021-01-15T18:40:20.186803+0000","last_undegraded":"2021-01-15T18:40:00.019710+0000","last_fullsized":"2021-01-15T18:39:50.717064+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1040,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8519680,"num_objects":1040,"num_object_clones":0,"num_object_copies":4160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":8320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1040}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"25'1063","reported_seq":"1109","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.189974+0000","last_change":"2021-01-15T18:40:12.041041+0000","last_active":"2021-01-15T18:40:20.189974+0000","last_peered":"2021-01-15T18:40:20.189974+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:39:50.728122+0000","last_became_peered":"2021-01-15T18:39:50.728122+0000","last_unstale":"2021-01-15T18:40:20.189974+0000","last_undegraded":"2021-01-15T18:39:50.722745+0000","last_fullsized":"2021-01-15T18:39:50.722291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1063,"ondisk_log_size":1063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8659110,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1061,"num_write_kb":8468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"25'982","reported_seq":"1043","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.104234+0000","last_change":"2021-01-15T18:40:12.764136+0000","last_active":"2021-01-15T18:40:20.104234+0000","last_peered":"2021-01-15T18:40:20.104234+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:39:50.730670+0000","last_became_peered":"2021-01-15T18:39:50.730670+0000","last_unstale":"2021-01-15T18:40:20.104234+0000","last_undegraded":"2021-01-15T18:39:50.716754+0000","last_fullsized":"2021-01-15T18:39:50.716267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":982,"ondisk_log_size":982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8011942,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":980,"num_write_kb":7831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"25'994","reported_seq":"1029","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.792293+0000","last_change":"2021-01-15T18:40:13.513752+0000","last_active":"2021-01-15T18:40:20.792293+0000","last_peered":"2021-01-15T18:40:20.792293+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:39:50.737732+0000","last_became_peered":"2021-01-15T18:39:50.737732+0000","last_unstale":"2021-01-15T18:40:20.792293+0000","last_undegraded":"2021-01-15T18:39:50.718708+0000","last_fullsized":"2021-01-15T18:39:50.718303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":994,"ondisk_log_size":994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8102054,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":992,"num_write_kb":7924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"25'1047","reported_seq":"1104","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.123402+0000","last_change":"2021-01-15T18:40:14.130508+0000","last_active":"2021-01-15T18:40:20.123402+0000","last_peered":"2021-01-15T18:40:20.123402+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:39:50.737428+0000","last_became_peered":"2021-01-15T18:39:50.737428+0000","last_unstale":"2021-01-15T18:40:20.123402+0000","last_undegraded":"2021-01-15T18:39:50.718079+0000","last_fullsized":"2021-01-15T18:39:50.716235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1047,"ondisk_log_size":1047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8564819,"num_objects":1046,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1046,"num_write_kb":8365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1046}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"25'1033","reported_seq":"1056","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.217346+0000","last_change":"2021-01-15T18:40:14.677677+0000","last_active":"2021-01-15T18:40:20.217346+0000","last_peered":"2021-01-15T18:40:20.217346+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:39:50.742321+0000","last_became_peered":"2021-01-15T18:39:50.742321+0000","last_unstale":"2021-01-15T18:40:20.217346+0000","last_undegraded":"2021-01-15T18:39:50.718397+0000","last_fullsized":"2021-01-15T18:39:50.718163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1033,"ondisk_log_size":1033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8450131,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1032,"num_write_kb":8253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"25'1022","reported_seq":"1061","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.210517+0000","last_change":"2021-01-15T18:40:15.497870+0000","last_active":"2021-01-15T18:40:20.210517+0000","last_peered":"2021-01-15T18:40:20.210517+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:39:50.730601+0000","last_became_peered":"2021-01-15T18:39:50.730601+0000","last_unstale":"2021-01-15T18:40:20.210517+0000","last_undegraded":"2021-01-15T18:39:50.717607+0000","last_fullsized":"2021-01-15T18:39:50.717272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1022,"ondisk_log_size":1022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8319225,"num_objects":1017,"num_object_clones":0,"num_object_copies":4068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1019,"num_write_kb":8132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"25'967","reported_seq":"1004","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.197509+0000","last_change":"2021-01-15T18:40:16.127785+0000","last_active":"2021-01-15T18:40:20.197509+0000","last_peered":"2021-01-15T18:40:20.197509+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:39:50.738893+0000","last_became_peered":"2021-01-15T18:39:50.738893+0000","last_unstale":"2021-01-15T18:40:20.197509+0000","last_undegraded":"2021-01-15T18:39:50.716614+0000","last_fullsized":"2021-01-15T18:39:50.715961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":967,"ondisk_log_size":967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7889062,"num_objects":964,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":965,"num_write_kb":7711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'965","reported_seq":"1011","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.201446+0000","last_change":"2021-01-15T18:40:16.596012+0000","last_active":"2021-01-15T18:40:20.201446+0000","last_peered":"2021-01-15T18:40:20.201446+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:39:50.735502+0000","last_became_peered":"2021-01-15T18:39:50.735502+0000","last_unstale":"2021-01-15T18:40:20.201446+0000","last_undegraded":"2021-01-15T18:39:50.727060+0000","last_fullsized":"2021-01-15T18:39:50.726618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":965,"ondisk_log_size":965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7864486,"num_objects":961,"num_object_clones":0,"num_object_copies":3844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":963,"num_write_kb":7687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":961}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"25'972","reported_seq":"1018","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.190075+0000","last_change":"2021-01-15T18:40:17.426018+0000","last_active":"2021-01-15T18:40:20.190075+0000","last_peered":"2021-01-15T18:40:20.190075+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:39:50.735947+0000","last_became_peered":"2021-01-15T18:39:50.735947+0000","last_unstale":"2021-01-15T18:40:20.190075+0000","last_undegraded":"2021-01-15T18:39:50.725685+0000","last_fullsized":"2021-01-15T18:39:50.725245+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":972,"ondisk_log_size":972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7905446,"num_objects":966,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":970,"num_write_kb":7732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"25'1046","reported_seq":"1092","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.180671+0000","last_change":"2021-01-15T18:40:18.167824+0000","last_active":"2021-01-15T18:40:20.180671+0000","last_peered":"2021-01-15T18:40:20.180671+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:39:50.731549+0000","last_became_peered":"2021-01-15T18:39:50.731549+0000","last_unstale":"2021-01-15T18:40:20.180671+0000","last_undegraded":"2021-01-15T18:39:50.723705+0000","last_fullsized":"2021-01-15T18:39:50.723244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1046,"ondisk_log_size":1046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8519846,"num_objects":1041,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":8332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"25'1040","reported_seq":"1079","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.790085+0000","last_change":"2021-01-15T18:40:18.900181+0000","last_active":"2021-01-15T18:40:20.790085+0000","last_peered":"2021-01-15T18:40:20.790085+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:39:50.739488+0000","last_became_peered":"2021-01-15T18:39:50.739488+0000","last_unstale":"2021-01-15T18:40:20.790085+0000","last_undegraded":"2021-01-15T18:39:50.716988+0000","last_fullsized":"2021-01-15T18:39:50.716568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1040,"ondisk_log_size":1040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8438092,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1037,"num_write_kb":8259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"25'981","reported_seq":"1041","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:20.123630+0000","last_change":"2021-01-15T18:40:19.523735+0000","last_active":"2021-01-15T18:40:20.123630+0000","last_peered":"2021-01-15T18:40:20.123630+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:39:50.731209+0000","last_became_peered":"2021-01-15T18:39:50.731209+0000","last_unstale":"2021-01-15T18:40:20.123630+0000","last_undegraded":"2021-01-15T18:39:50.717301+0000","last_fullsized":"2021-01-15T18:39:50.716845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":981,"ondisk_log_size":981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8011942,"num_objects":979,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":979,"num_write_kb":7826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":131762452,"num_objects":16098,"num_object_clones":0,"num_object_copies":64392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16125,"num_write_kb":128790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":197660672,"data_stored":197660672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16152,"ondisk_log_size":16152,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215114,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106724,"kb_used_data":64412,"kb_used_omap":0,"kb_used_meta":42303,"kb_avail":94265116,"statfs":{"total":96636764160,"available":96527478784,"internally_reserved":0,"allocated":65957888,"data_stored":65813097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":43318324},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,10,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8170000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1619999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109728,"kb_used_data":63576,"kb_used_omap":0,"kb_used_meta":46144,"kb_avail":94262112,"statfs":{"total":96636764160,"available":96524402688,"internally_reserved":0,"allocated":65101824,"data_stored":64957033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47251456},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4319999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110924,"kb_used_data":64452,"kb_used_omap":0,"kb_used_meta":46463,"kb_avail":94260916,"statfs":{"total":96636764160,"available":96523177984,"internally_reserved":0,"allocated":65998848,"data_stored":65854057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":47578164},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5276,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366564,"statfs":{"total":96636764160,"available":96631361536,"internally_reserved":0,"allocated":217088,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2021-01-15T18:40:24.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:24.790 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:24.792 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:24.802 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2021-01-15T18:40:24.369137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149104916,"num_objects":18215,"num_object_clones":0,"num_object_copies":72860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18242,"num_write_kb":145726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18269,"ondisk_log_size":18269,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":362140,"kb_used_data":215928,"kb_used_omap":1,"kb_used_meta":146174,"kb_avail":377125220,"statfs":{"total":386547056640,"available":386176225280,"internally_reserved":0,"allocated":221110272,"data_stored":220560536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1944,"internal_metadata":149682280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,10,14,7,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":86589440,"num_objects":10570,"num_object_clones":0,"num_object_copies":42280,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":10516,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":10570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":84560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006717"},"pg_stats":[{"pgid":"1.f","version":"25'1107","reported_seq":"1133","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.099781+0000","last_change":"2021-01-15T18:40:00.048487+0000","last_active":"2021-01-15T18:40:24.099781+0000","last_peered":"2021-01-15T18:40:24.099781+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.742032+0000","last_became_peered":"2021-01-15T18:39:50.742032+0000","last_unstale":"2021-01-15T18:40:24.099781+0000","last_undegraded":"2021-01-15T18:40:00.031609+0000","last_fullsized":"2021-01-15T18:39:50.716998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1107,"ondisk_log_size":1107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9068544,"num_objects":1107,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":8856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"25'1142","reported_seq":"1188","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.446272+0000","last_change":"2021-01-15T18:40:10.885681+0000","last_active":"2021-01-15T18:40:23.446272+0000","last_peered":"2021-01-15T18:40:23.446272+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:39:50.733030+0000","last_became_peered":"2021-01-15T18:39:50.733030+0000","last_unstale":"2021-01-15T18:40:23.446272+0000","last_undegraded":"2021-01-15T18:39:50.724613+0000","last_fullsized":"2021-01-15T18:39:50.724211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1142,"ondisk_log_size":1142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9314470,"num_objects":1138,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1140,"num_write_kb":9108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"25'1146","reported_seq":"1204","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.030106+0000","last_change":"2021-01-15T18:40:11.724276+0000","last_active":"2021-01-15T18:40:23.030106+0000","last_peered":"2021-01-15T18:40:23.030106+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:39:50.729885+0000","last_became_peered":"2021-01-15T18:39:50.729885+0000","last_unstale":"2021-01-15T18:40:23.030106+0000","last_undegraded":"2021-01-15T18:39:50.716824+0000","last_fullsized":"2021-01-15T18:39:50.716337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1146,"ondisk_log_size":1146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9359443,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1145,"num_write_kb":9146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"25'1176","reported_seq":"1202","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.106925+0000","last_change":"2021-01-15T18:40:00.064227+0000","last_active":"2021-01-15T18:40:24.106925+0000","last_peered":"2021-01-15T18:40:24.106925+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:39:50.739712+0000","last_became_peered":"2021-01-15T18:39:50.739712+0000","last_unstale":"2021-01-15T18:40:24.106925+0000","last_undegraded":"2021-01-15T18:40:00.019710+0000","last_fullsized":"2021-01-15T18:39:50.717064+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1176,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9633792,"num_objects":1176,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1176,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"25'1193","reported_seq":"1239","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.399047+0000","last_change":"2021-01-15T18:40:12.041041+0000","last_active":"2021-01-15T18:40:23.399047+0000","last_peered":"2021-01-15T18:40:23.399047+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:39:50.728122+0000","last_became_peered":"2021-01-15T18:39:50.728122+0000","last_unstale":"2021-01-15T18:40:23.399047+0000","last_undegraded":"2021-01-15T18:39:50.722745+0000","last_fullsized":"2021-01-15T18:39:50.722291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1193,"ondisk_log_size":1193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9724070,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1188}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"25'1099","reported_seq":"1160","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.018488+0000","last_change":"2021-01-15T18:40:12.764136+0000","last_active":"2021-01-15T18:40:23.018488+0000","last_peered":"2021-01-15T18:40:23.018488+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:39:50.730670+0000","last_became_peered":"2021-01-15T18:39:50.730670+0000","last_unstale":"2021-01-15T18:40:23.018488+0000","last_undegraded":"2021-01-15T18:39:50.716754+0000","last_fullsized":"2021-01-15T18:39:50.716267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1099,"ondisk_log_size":1099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8970406,"num_objects":1096,"num_object_clones":0,"num_object_copies":4384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":8767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"25'1124","reported_seq":"1159","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.123053+0000","last_change":"2021-01-15T18:40:13.513752+0000","last_active":"2021-01-15T18:40:24.123053+0000","last_peered":"2021-01-15T18:40:24.123053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:39:50.737732+0000","last_became_peered":"2021-01-15T18:39:50.737732+0000","last_unstale":"2021-01-15T18:40:24.123053+0000","last_undegraded":"2021-01-15T18:39:50.718708+0000","last_fullsized":"2021-01-15T18:39:50.718303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1124,"ondisk_log_size":1124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9167014,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":8964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"25'1158","reported_seq":"1215","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.027355+0000","last_change":"2021-01-15T18:40:14.130508+0000","last_active":"2021-01-15T18:40:23.027355+0000","last_peered":"2021-01-15T18:40:23.027355+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:39:50.737428+0000","last_became_peered":"2021-01-15T18:39:50.737428+0000","last_unstale":"2021-01-15T18:40:23.027355+0000","last_undegraded":"2021-01-15T18:39:50.718079+0000","last_fullsized":"2021-01-15T18:39:50.716235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1158,"ondisk_log_size":1158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9474131,"num_objects":1157,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1157,"num_write_kb":9253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1157}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"25'1192","reported_seq":"1215","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.109120+0000","last_change":"2021-01-15T18:40:14.677677+0000","last_active":"2021-01-15T18:40:24.109120+0000","last_peered":"2021-01-15T18:40:24.109120+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:39:50.742321+0000","last_became_peered":"2021-01-15T18:39:50.742321+0000","last_unstale":"2021-01-15T18:40:24.109120+0000","last_undegraded":"2021-01-15T18:39:50.718397+0000","last_fullsized":"2021-01-15T18:39:50.718163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1192,"ondisk_log_size":1192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9752659,"num_objects":1191,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"25'1178","reported_seq":"1217","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.115512+0000","last_change":"2021-01-15T18:40:15.497870+0000","last_active":"2021-01-15T18:40:24.115512+0000","last_peered":"2021-01-15T18:40:24.115512+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:39:50.730601+0000","last_became_peered":"2021-01-15T18:39:50.730601+0000","last_unstale":"2021-01-15T18:40:24.115512+0000","last_undegraded":"2021-01-15T18:39:50.717607+0000","last_fullsized":"2021-01-15T18:39:50.717272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1178,"ondisk_log_size":1178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9597177,"num_objects":1173,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1175,"num_write_kb":9380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"25'1117","reported_seq":"1154","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.011493+0000","last_change":"2021-01-15T18:40:16.127785+0000","last_active":"2021-01-15T18:40:24.011493+0000","last_peered":"2021-01-15T18:40:24.011493+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:39:50.738893+0000","last_became_peered":"2021-01-15T18:39:50.738893+0000","last_unstale":"2021-01-15T18:40:24.011493+0000","last_undegraded":"2021-01-15T18:39:50.716614+0000","last_fullsized":"2021-01-15T18:39:50.715961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1117,"ondisk_log_size":1117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9117862,"num_objects":1114,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1115,"num_write_kb":8911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'1096","reported_seq":"1142","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.447203+0000","last_change":"2021-01-15T18:40:16.596012+0000","last_active":"2021-01-15T18:40:23.447203+0000","last_peered":"2021-01-15T18:40:23.447203+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:39:50.735502+0000","last_became_peered":"2021-01-15T18:39:50.735502+0000","last_unstale":"2021-01-15T18:40:23.447203+0000","last_undegraded":"2021-01-15T18:39:50.727060+0000","last_fullsized":"2021-01-15T18:39:50.726618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1096,"ondisk_log_size":1096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8937638,"num_objects":1092,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1094,"num_write_kb":8735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"25'1078","reported_seq":"1124","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.464968+0000","last_change":"2021-01-15T18:40:17.426018+0000","last_active":"2021-01-15T18:40:23.464968+0000","last_peered":"2021-01-15T18:40:23.464968+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:39:50.735947+0000","last_became_peered":"2021-01-15T18:39:50.735947+0000","last_unstale":"2021-01-15T18:40:23.464968+0000","last_undegraded":"2021-01-15T18:39:50.725685+0000","last_fullsized":"2021-01-15T18:39:50.725245+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1078,"ondisk_log_size":1078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8773798,"num_objects":1072,"num_object_clones":0,"num_object_copies":4288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1076,"num_write_kb":8580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1072}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"25'1173","reported_seq":"1219","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.400209+0000","last_change":"2021-01-15T18:40:18.167824+0000","last_active":"2021-01-15T18:40:23.400209+0000","last_peered":"2021-01-15T18:40:23.400209+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:39:50.731549+0000","last_became_peered":"2021-01-15T18:39:50.731549+0000","last_unstale":"2021-01-15T18:40:23.400209+0000","last_undegraded":"2021-01-15T18:39:50.723705+0000","last_fullsized":"2021-01-15T18:39:50.723244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1173,"ondisk_log_size":1173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9560230,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":9348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"25'1205","reported_seq":"1244","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:24.107043+0000","last_change":"2021-01-15T18:40:18.900181+0000","last_active":"2021-01-15T18:40:24.107043+0000","last_peered":"2021-01-15T18:40:24.107043+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:39:50.739488+0000","last_became_peered":"2021-01-15T18:39:50.739488+0000","last_unstale":"2021-01-15T18:40:24.107043+0000","last_undegraded":"2021-01-15T18:39:50.716988+0000","last_fullsized":"2021-01-15T18:39:50.716568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1205,"ondisk_log_size":1205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9789772,"num_objects":1197,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":9579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"25'1085","reported_seq":"1145","reported_epoch":"25","state":"active+undersized+degraded","last_fresh":"2021-01-15T18:40:23.005408+0000","last_change":"2021-01-15T18:40:19.523735+0000","last_active":"2021-01-15T18:40:23.005408+0000","last_peered":"2021-01-15T18:40:23.005408+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:39:50.731209+0000","last_became_peered":"2021-01-15T18:39:50.731209+0000","last_unstale":"2021-01-15T18:40:23.005408+0000","last_undegraded":"2021-01-15T18:39:50.717301+0000","last_fullsized":"2021-01-15T18:39:50.716845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1085,"ondisk_log_size":1085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8863910,"num_objects":1083,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":8658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1083}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149104916,"num_objects":18215,"num_object_clones":0,"num_object_copies":72860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18242,"num_write_kb":145726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222797824,"data_stored":222797824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18269,"ondisk_log_size":18269,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215115,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121632,"kb_used_data":71892,"kb_used_omap":0,"kb_used_meta":49727,"kb_avail":94250208,"statfs":{"total":96636764160,"available":96512212992,"internally_reserved":0,"allocated":73617408,"data_stored":73472617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":50920500},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,4,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125080,"kb_used_data":71376,"kb_used_omap":0,"kb_used_meta":53696,"kb_avail":94246760,"statfs":{"total":96636764160,"available":96508682240,"internally_reserved":0,"allocated":73089024,"data_stored":72944233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54984704},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4319999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110152,"kb_used_data":72448,"kb_used_omap":0,"kb_used_meta":37695,"kb_avail":94261688,"statfs":{"total":96636764160,"available":96523968512,"internally_reserved":0,"allocated":74186752,"data_stored":74041961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":38599732},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0129999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5276,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366564,"statfs":{"total":96636764160,"available":96631361536,"internally_reserved":0,"allocated":217088,"data_stored":101725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2021-01-15T18:40:24.804 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2021-01-15T18:40:24.804 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2021-01-15T18:40:24.804 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2021-01-15T18:40:24.883 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2021-01-15T18:40:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:24.895 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2021-01-15T18:40:24.896 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2021-01-15T18:40:25.002 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2021-01-15T18:40:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.005 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2021-01-15T18:40:25.006 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2021-01-15T18:40:25.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 25 20 18850 18830 5.88318 3.92969 0.0249055 0.0265453 2021-01-15T18:40:25.109 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2021-01-15T18:40:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.123 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2021-01-15T18:40:25.123 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2021-01-15T18:40:25.291 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2021-01-15T18:40:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.300 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2021-01-15T18:40:25.300 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2021-01-15T18:40:25.430 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2021-01-15T18:40:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.436 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2021-01-15T18:40:25.437 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2021-01-15T18:40:25.532 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2021-01-15T18:40:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.542 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2021-01-15T18:40:25.543 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2021-01-15T18:40:25.613 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2021-01-15T18:40:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.616 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2021-01-15T18:40:25.617 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2021-01-15T18:40:25.656 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2021-01-15T18:40:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.658 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2021-01-15T18:40:25.659 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2021-01-15T18:40:25.738 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2021-01-15T18:40:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.740 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2021-01-15T18:40:25.741 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2021-01-15T18:40:25.934 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2021-01-15T18:40:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:25.951 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2021-01-15T18:40:25.951 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2021-01-15T18:40:26.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 26 20 19478 19458 5.84559 4.90625 0.0279259 0.0267195 2021-01-15T18:40:26.047 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2021-01-15T18:40:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.053 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2021-01-15T18:40:26.054 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2021-01-15T18:40:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2021-01-15T18:40:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.152 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2021-01-15T18:40:26.152 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2021-01-15T18:40:26.246 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2021-01-15T18:40:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.250 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2021-01-15T18:40:26.250 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2021-01-15T18:40:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2021-01-15T18:40:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.404 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2021-01-15T18:40:26.404 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2021-01-15T18:40:26.537 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2021-01-15T18:40:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.540 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2021-01-15T18:40:26.541 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2021-01-15T18:40:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2021-01-15T18:40:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.673 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2021-01-15T18:40:26.673 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2021-01-15T18:40:26.810 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2021-01-15T18:40:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.813 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2021-01-15T18:40:26.814 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2021-01-15T18:40:26.911 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2021-01-15T18:40:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.915 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2021-01-15T18:40:26.916 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2021-01-15T18:40:26.972 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2021-01-15T18:40:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:26.974 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2021-01-15T18:40:26.974 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2021-01-15T18:40:27.009 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 27 20 20154 20134 5.82467 5.28125 0.00504487 0.0266981 2021-01-15T18:40:27.013 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2021-01-15T18:40:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.015 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2021-01-15T18:40:27.016 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2021-01-15T18:40:27.137 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2021-01-15T18:40:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.146 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2021-01-15T18:40:27.146 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2021-01-15T18:40:27.268 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2021-01-15T18:40:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.271 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2021-01-15T18:40:27.271 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2021-01-15T18:40:27.369 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2021-01-15T18:40:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.373 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2021-01-15T18:40:27.373 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2021-01-15T18:40:27.489 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2021-01-15T18:40:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.500 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2021-01-15T18:40:27.500 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2021-01-15T18:40:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2021-01-15T18:40:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.598 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2021-01-15T18:40:27.599 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2021-01-15T18:40:27.707 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2021-01-15T18:40:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.711 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2021-01-15T18:40:27.711 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2021-01-15T18:40:27.814 INFO:teuthology.orchestra.run.smithi148.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2021-01-15T18:40:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T18:40:27.818 INFO:tasks.ceph.osd.1:Restarting daemon 2021-01-15T18:40:27.818 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-01-15T18:40:27.821 INFO:tasks.ceph.osd.1:Started 2021-01-15T18:40:27.821 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-01-15T18:40:28.010 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 28 20 20827 20807 5.80429 5.25781 0.0293501 0.0269031 2021-01-15T18:40:28.093 INFO:teuthology.orchestra.run.smithi148.stderr:no valid command found; 10 closest matches: 2021-01-15T18:40:28.093 INFO:teuthology.orchestra.run.smithi148.stderr:0 2021-01-15T18:40:28.094 INFO:teuthology.orchestra.run.smithi148.stderr:1 2021-01-15T18:40:28.094 INFO:teuthology.orchestra.run.smithi148.stderr:2 2021-01-15T18:40:28.094 INFO:teuthology.orchestra.run.smithi148.stderr:abort 2021-01-15T18:40:28.094 INFO:teuthology.orchestra.run.smithi148.stderr:assert 2021-01-15T18:40:28.094 INFO:teuthology.orchestra.run.smithi148.stderr:config diff 2021-01-15T18:40:28.095 INFO:teuthology.orchestra.run.smithi148.stderr:config diff get 2021-01-15T18:40:28.095 INFO:teuthology.orchestra.run.smithi148.stderr:config get 2021-01-15T18:40:28.095 INFO:teuthology.orchestra.run.smithi148.stderr:config help [] 2021-01-15T18:40:28.095 INFO:teuthology.orchestra.run.smithi148.stderr:config set ... 2021-01-15T18:40:28.095 INFO:teuthology.orchestra.run.smithi148.stderr:admin_socket: invalid command 2021-01-15T18:40:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-01-15T18:40:28.097 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2021-01-15T18:40:28.250 INFO:tasks.ceph.osd.1.smithi148.stderr:2021-01-15T18:40:28.250+0000 7fd9c6e8e080 -1 Falling back to public interface 2021-01-15T18:40:29.010 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 29 20 21490 21470 5.78274 5.17969 0.0134007 0.0270034 2021-01-15T18:40:29.464 INFO:tasks.ceph.osd.1.smithi148.stderr:2021-01-15T18:40:29.463+0000 7fd9c6e8e080 -1 osd.1 19 log_to_monitors {default=true} 2021-01-15T18:40:30.011 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 30 20 22278 22258 5.7951 6.15625 0.0217979 0.0269468 2021-01-15T18:40:31.011 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 31 20 22651 22631 5.70216 2.91406 0.0242921 0.0269224 2021-01-15T18:40:32.011 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 32 20 22651 22631 5.52397 0 - 0.0269224 2021-01-15T18:40:33.014 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 33 20 22760 22740 5.38186 0.425781 0.0578332 0.0289987 2021-01-15T18:40:33.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-01-15T18:40:33.536 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:40:33.536 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2021-01-15T18:40:33.536 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2021-01-15T18:40:33.537 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:40:33.567 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2021-01-15T18:40:33.568 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:40:34.014 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 34 20 23159 23139 5.31526 3.11719 0.0198089 0.0293115 2021-01-15T18:40:34.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:34.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:40:32.476691+0000","last_up_change":"2021-01-15T18:40:30.471863+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2021-01-15T18:39:24.508766+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":27,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":38392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":38392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":38392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":38392}]},"public_addr":"172.21.15.148:6804/38392","cluster_addr":"172.21.15.148:6805/38392","heartbeat_back_addr":"172.21.15.148:6807/38392","heartbeat_front_addr":"172.21.15.148:6806/38392","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":26,"up_thru":26,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":43629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":43629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":43629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":43629}]},"public_addr":"172.21.15.148:6812/43629","cluster_addr":"172.21.15.148:6813/43629","heartbeat_back_addr":"172.21.15.148:6815/43629","heartbeat_front_addr":"172.21.15.148:6814/43629","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":24,"up_thru":27,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":39252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":39252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":39252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":39252}]},"public_addr":"172.21.15.148:6808/39252","cluster_addr":"172.21.15.148:6809/39252","heartbeat_back_addr":"172.21.15.148:6811/39252","heartbeat_front_addr":"172.21.15.148:6810/39252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:40:34.406 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2021-01-15T18:40:34.407 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2021-01-15T18:40:34.407 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:35.014 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 35 20 23541 23521 5.24867 2.98438 0.0498187 0.0297422 2021-01-15T18:40:35.331 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:35.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2021-01-15T18:40:34.371880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188631316,"num_objects":23040,"num_object_clones":0,"num_object_copies":92160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23067,"num_write_kb":184326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23094,"ondisk_log_size":23094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":452244,"kb_used_data":275004,"kb_used_omap":2,"kb_used_meta":177213,"kb_avail":377035116,"statfs":{"total":386547056640,"available":386083958784,"internally_reserved":0,"allocated":281604096,"data_stored":281015039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":181466268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,3,0,4,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001791"},"pg_stats":[{"pgid":"1.f","version":"28'1384","reported_seq":"1422","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.015006+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:34.015006+0000","last_peered":"2021-01-15T18:40:34.015006+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:34.015006+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1384,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11337728,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":30}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1446","reported_seq":"1544","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.615801+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:33.615801+0000","last_peered":"2021-01-15T18:40:33.615801+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:33.615801+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1446,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11804838,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":45}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1434","reported_seq":"1505","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.071523+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:33.071523+0000","last_peered":"2021-01-15T18:40:33.071523+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:33.071523+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1434,"ondisk_log_size":1434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11718739,"num_objects":1431,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1433,"num_write_kb":11450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":15}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1466","reported_seq":"1504","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.016736+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:34.016736+0000","last_peered":"2021-01-15T18:40:34.016736+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:34.016736+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1466,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12009472,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1527","reported_seq":"1583","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.490773+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:33.490773+0000","last_peered":"2021-01-15T18:40:33.490773+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:33.490773+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1527,"ondisk_log_size":1527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12460198,"num_objects":1522,"num_object_clones":0,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":12180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1360","reported_seq":"1431","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.061423+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:33.061423+0000","last_peered":"2021-01-15T18:40:33.061423+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:33.061423+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1360,"ondisk_log_size":1360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11108518,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":10855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1430","reported_seq":"1476","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.971059+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:33.971059+0000","last_peered":"2021-01-15T18:40:33.971059+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:33.971059+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1430,"ondisk_log_size":1430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11673766,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":11412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":34}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1465","reported_seq":"1529","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.104030+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:33.104030+0000","last_peered":"2021-01-15T18:40:33.104030+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:33.104030+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1465,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11989075,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":11709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1474","reported_seq":"1508","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.973452+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:33.973452+0000","last_peered":"2021-01-15T18:40:33.973452+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:33.973452+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1474,"ondisk_log_size":1474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12062803,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1469","reported_seq":"1521","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.044839+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:34.044839+0000","last_peered":"2021-01-15T18:40:34.044839+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:34.044839+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1469,"ondisk_log_size":1469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11981049,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1415","reported_seq":"1465","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.137846+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:34.137846+0000","last_peered":"2021-01-15T18:40:34.137846+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:34.137846+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1415,"ondisk_log_size":1415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11559078,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":11295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":31}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1398","reported_seq":"1454","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.568883+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:33.568883+0000","last_peered":"2021-01-15T18:40:33.568883+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:33.568883+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1398,"ondisk_log_size":1398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11411622,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":11151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1429","reported_seq":"1487","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.385404+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:33.385404+0000","last_peered":"2021-01-15T18:40:33.385404+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:33.385404+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1429,"ondisk_log_size":1429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11649190,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":11388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":18}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1467","reported_seq":"1524","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.521479+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:33.521479+0000","last_peered":"2021-01-15T18:40:33.521479+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:33.521479+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1467,"ondisk_log_size":1467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11968678,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":11700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1536","reported_seq":"1588","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.043368+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:34.043368+0000","last_peered":"2021-01-15T18:40:34.043368+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:34.043368+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1536,"ondisk_log_size":1536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12501324,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1533,"num_write_kb":12227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":42}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1394","reported_seq":"1467","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.107590+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:33.107590+0000","last_peered":"2021-01-15T18:40:33.107590+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:33.107590+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1394,"ondisk_log_size":1394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11395238,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1392,"num_write_kb":11130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188631316,"num_objects":23040,"num_object_clones":0,"num_object_copies":92160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23067,"num_write_kb":184326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282742784,"data_stored":282742784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23094,"ondisk_log_size":23094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":91692,"kb_used_omap":0,"kb_used_meta":54655,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":93892608,"data_stored":93734710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55966772},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149432,"kb_used_data":90928,"kb_used_omap":0,"kb_used_meta":58496,"kb_avail":94222408,"statfs":{"total":96636764160,"available":96483745792,"internally_reserved":0,"allocated":93110272,"data_stored":92952374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59899904},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280526,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151056,"kb_used_data":92112,"kb_used_omap":0,"kb_used_meta":58943,"kb_avail":94220784,"statfs":{"total":96636764160,"available":96482082816,"internally_reserved":0,"allocated":94322688,"data_stored":94164790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60357684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.311}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.705}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.362}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":5119,"kb_avail":94366440,"statfs":{"total":96636764160,"available":96631234560,"internally_reserved":0,"allocated":278528,"data_stored":163165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":5241908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]}]}} 2021-01-15T18:40:35.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:36.015 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 36 20 23910 23890 5.18287 2.88281 0.0651835 0.0301216 2021-01-15T18:40:36.215 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:36.249 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:36.277 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2021-01-15T18:40:34.371880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188631316,"num_objects":23040,"num_object_clones":0,"num_object_copies":92160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23067,"num_write_kb":184326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23094,"ondisk_log_size":23094,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":452244,"kb_used_data":275004,"kb_used_omap":2,"kb_used_meta":177213,"kb_avail":377035116,"statfs":{"total":386547056640,"available":386083958784,"internally_reserved":0,"allocated":281604096,"data_stored":281015039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":181466268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,3,0,4,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33644544,"num_objects":4107,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001791"},"pg_stats":[{"pgid":"1.f","version":"28'1384","reported_seq":"1422","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.015006+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:34.015006+0000","last_peered":"2021-01-15T18:40:34.015006+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:34.015006+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1384,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11337728,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":30}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1446","reported_seq":"1544","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.615801+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:33.615801+0000","last_peered":"2021-01-15T18:40:33.615801+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:33.615801+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1446,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11804838,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":45}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1434","reported_seq":"1505","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.071523+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:33.071523+0000","last_peered":"2021-01-15T18:40:33.071523+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:33.071523+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1434,"ondisk_log_size":1434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11718739,"num_objects":1431,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1433,"num_write_kb":11450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":15}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1466","reported_seq":"1504","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.016736+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:34.016736+0000","last_peered":"2021-01-15T18:40:34.016736+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:34.016736+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1466,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12009472,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1527","reported_seq":"1583","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.490773+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:33.490773+0000","last_peered":"2021-01-15T18:40:33.490773+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:33.490773+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1527,"ondisk_log_size":1527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12460198,"num_objects":1522,"num_object_clones":0,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":12180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1360","reported_seq":"1431","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.061423+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:33.061423+0000","last_peered":"2021-01-15T18:40:33.061423+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:33.061423+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1360,"ondisk_log_size":1360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11108518,"num_objects":1357,"num_object_clones":0,"num_object_copies":5428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":10855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1430","reported_seq":"1476","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.971059+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:33.971059+0000","last_peered":"2021-01-15T18:40:33.971059+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:33.971059+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1430,"ondisk_log_size":1430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11673766,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":11412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":34}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1465","reported_seq":"1529","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.104030+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:33.104030+0000","last_peered":"2021-01-15T18:40:33.104030+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:33.104030+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1465,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11989075,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":11709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1474","reported_seq":"1508","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.973452+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:33.973452+0000","last_peered":"2021-01-15T18:40:33.973452+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:33.973452+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1474,"ondisk_log_size":1474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12062803,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1469","reported_seq":"1521","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.044839+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:34.044839+0000","last_peered":"2021-01-15T18:40:34.044839+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:34.044839+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1469,"ondisk_log_size":1469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11981049,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1415","reported_seq":"1465","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.137846+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:34.137846+0000","last_peered":"2021-01-15T18:40:34.137846+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:34.137846+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1415,"ondisk_log_size":1415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11559078,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":11295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":31}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1398","reported_seq":"1454","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.568883+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:33.568883+0000","last_peered":"2021-01-15T18:40:33.568883+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:33.568883+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1398,"ondisk_log_size":1398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11411622,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":11151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1429","reported_seq":"1487","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.385404+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:33.385404+0000","last_peered":"2021-01-15T18:40:33.385404+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:33.385404+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1429,"ondisk_log_size":1429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11649190,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":11388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":18}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1467","reported_seq":"1524","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.521479+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:33.521479+0000","last_peered":"2021-01-15T18:40:33.521479+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:33.521479+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1467,"ondisk_log_size":1467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11968678,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":11700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1536","reported_seq":"1588","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.043368+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:34.043368+0000","last_peered":"2021-01-15T18:40:34.043368+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:34.043368+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1536,"ondisk_log_size":1536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12501324,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1533,"num_write_kb":12227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":42}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1394","reported_seq":"1467","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.107590+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:33.107590+0000","last_peered":"2021-01-15T18:40:33.107590+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:33.107590+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1394,"ondisk_log_size":1394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11395238,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1392,"num_write_kb":11130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":188631316,"num_objects":23040,"num_object_clones":0,"num_object_copies":92160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23067,"num_write_kb":184326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282742784,"data_stored":282742784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23094,"ondisk_log_size":23094,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":91692,"kb_used_omap":0,"kb_used_meta":54655,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":93892608,"data_stored":93734710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55966772},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149432,"kb_used_data":90928,"kb_used_omap":0,"kb_used_meta":58496,"kb_avail":94222408,"statfs":{"total":96636764160,"available":96483745792,"internally_reserved":0,"allocated":93110272,"data_stored":92952374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59899904},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280526,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151056,"kb_used_data":92112,"kb_used_omap":0,"kb_used_meta":58943,"kb_avail":94220784,"statfs":{"total":96636764160,"available":96482082816,"internally_reserved":0,"allocated":94322688,"data_stored":94164790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60357684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.311}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.705}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.362}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":5119,"kb_avail":94366440,"statfs":{"total":96636764160,"available":96631234560,"internally_reserved":0,"allocated":278528,"data_stored":163165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":5241908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]}]}} 2021-01-15T18:40:36.278 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:40:37.015 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 37 20 24294 24274 5.12388 3 0.0229821 0.0304678 2021-01-15T18:40:37.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:37.253 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22647/92160 objects degraded (24.574%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":87,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23040,"data_bytes":188631316,"bytes_used":482766848,"bytes_avail":386064289792,"bytes_total":386547056640,"degraded_objects":22647,"degraded_total":92160,"degraded_ratio":0.24573567708333333,"recovering_objects_per_sec":3,"recovering_bytes_per_sec":28663,"recovering_keys_per_sec":0,"num_objects_recovered":21,"num_bytes_recovered":172032,"num_keys_recovered":0,"write_bytes_sec":2314947,"read_op_per_sec":0,"write_op_per_sec":282},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2021-01-15T18:40:14.363153+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:40:37.318 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:40:37.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:38.015 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 38 20 24679 24659 5.0682 3.00781 0.0383866 0.0308044 2021-01-15T18:40:38.484 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:38.515 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2021-01-15T18:40:38.372870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":192538900,"num_objects":23517,"num_object_clones":0,"num_object_copies":94068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23544,"num_write_kb":188142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23571,"ondisk_log_size":23571,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":470536,"kb_used_data":285616,"kb_used_omap":2,"kb_used_meta":184893,"kb_avail":377016824,"statfs":{"total":386547056640,"available":386065227776,"internally_reserved":0,"allocated":292470784,"data_stored":291839192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":189330588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,6,5,4,4,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17801216,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.891319"},"pg_stats":[{"pgid":"1.f","version":"28'1384","reported_seq":"1422","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.015006+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:34.015006+0000","last_peered":"2021-01-15T18:40:34.015006+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:34.015006+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1384,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11337728,"num_objects":1384,"num_object_clones":0,"num_object_copies":5536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":30}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1446","reported_seq":"1544","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.615801+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:33.615801+0000","last_peered":"2021-01-15T18:40:33.615801+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:33.615801+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1446,"ondisk_log_size":1446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11804838,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":11540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":45}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1546","reported_seq":"1616","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:37.983962+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:37.983962+0000","last_peered":"2021-01-15T18:40:37.983962+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:37.983962+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1546,"ondisk_log_size":1546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12636243,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":12346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1466","reported_seq":"1504","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.016736+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:34.016736+0000","last_peered":"2021-01-15T18:40:34.016736+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:34.016736+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1466,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12009472,"num_objects":1466,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1527","reported_seq":"1583","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.490773+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:33.490773+0000","last_peered":"2021-01-15T18:40:33.490773+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:33.490773+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1527,"ondisk_log_size":1527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12460198,"num_objects":1522,"num_object_clones":0,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":12180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1475","reported_seq":"1546","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.096931+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:38.096931+0000","last_peered":"2021-01-15T18:40:38.096931+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:38.096931+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1475,"ondisk_log_size":1475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12050598,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":127}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1430","reported_seq":"1476","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.971059+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:33.971059+0000","last_peered":"2021-01-15T18:40:33.971059+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:33.971059+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1430,"ondisk_log_size":1430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11673766,"num_objects":1426,"num_object_clones":0,"num_object_copies":5704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":11412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":34}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1582","reported_seq":"1648","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.038994+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:38.038994+0000","last_peered":"2021-01-15T18:40:38.038994+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:38.038994+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1582,"ondisk_log_size":1582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12947539,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1581,"num_write_kb":12645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1474","reported_seq":"1508","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.973452+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:33.973452+0000","last_peered":"2021-01-15T18:40:33.973452+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:33.973452+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1474,"ondisk_log_size":1474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12062803,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":37}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1469","reported_seq":"1521","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.044839+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:34.044839+0000","last_peered":"2021-01-15T18:40:34.044839+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:34.044839+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1469,"ondisk_log_size":1469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11981049,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":11708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1415","reported_seq":"1465","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.137846+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:34.137846+0000","last_peered":"2021-01-15T18:40:34.137846+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:34.137846+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1415,"ondisk_log_size":1415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11559078,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1413,"num_write_kb":11295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":31}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1398","reported_seq":"1454","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.568883+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:33.568883+0000","last_peered":"2021-01-15T18:40:33.568883+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:33.568883+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1398,"ondisk_log_size":1398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11411622,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":11151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1429","reported_seq":"1487","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.385404+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:33.385404+0000","last_peered":"2021-01-15T18:40:33.385404+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:33.385404+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1429,"ondisk_log_size":1429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11649190,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":11388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":18}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1467","reported_seq":"1524","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:33.521479+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:33.521479+0000","last_peered":"2021-01-15T18:40:33.521479+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:33.521479+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1467,"ondisk_log_size":1467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11968678,"num_objects":1462,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":11700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1536","reported_seq":"1588","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:34.043368+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:34.043368+0000","last_peered":"2021-01-15T18:40:34.043368+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:34.043368+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1536,"ondisk_log_size":1536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12501324,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1533,"num_write_kb":12227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":42}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1527","reported_seq":"1600","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.034347+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:38.034347+0000","last_peered":"2021-01-15T18:40:38.034347+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:38.034347+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1527,"ondisk_log_size":1527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12484774,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":12194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":192538900,"num_objects":23517,"num_object_clones":0,"num_object_copies":94068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23544,"num_write_kb":188142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":172032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":293531648,"data_stored":293531648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23571,"ondisk_log_size":23571,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215117,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":91692,"kb_used_omap":0,"kb_used_meta":54655,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":93892608,"data_stored":93734710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55966772},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148516,"kb_used_data":98716,"kb_used_omap":0,"kb_used_meta":49792,"kb_avail":94223324,"statfs":{"total":96636764160,"available":96484683776,"internally_reserved":0,"allocated":101085184,"data_stored":100927286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50987008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280526,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151056,"kb_used_data":92112,"kb_used_omap":0,"kb_used_meta":58943,"kb_avail":94220784,"statfs":{"total":96636764160,"available":96482082816,"internally_reserved":0,"allocated":94322688,"data_stored":94164790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60357684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.311}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.705}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.362}]}]},{"osd":1,"up_from":26,"seq":111669149698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24608,"kb_used_data":3096,"kb_used_omap":0,"kb_used_meta":21503,"kb_avail":94347232,"statfs":{"total":96636764160,"available":96611565568,"internally_reserved":0,"allocated":3170304,"data_stored":3012406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":22019124},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]}]}]}} 2021-01-15T18:40:39.016 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 39 20 25095 25075 5.02147 3.25 0.061265 0.0310939 2021-01-15T18:40:40.017 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:40:40.017942+0000 min lat: 0.00287162 max lat: 2.35257 avg lat: 0.0313437 2021-01-15T18:40:40.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:40:40.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 40 20 25533 25513 4.98137 3.42188 0.0483015 0.0313437 2021-01-15T18:40:41.017 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 41 20 25948 25928 4.93895 3.24219 0.0500818 0.0316131 2021-01-15T18:40:41.518 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:42.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 42 20 26356 26336 4.89726 3.1875 0.0689932 0.0318845 2021-01-15T18:40:42.633 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:42.641 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:42.667 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2021-01-15T18:40:42.373809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":204654868,"num_objects":24996,"num_object_clones":0,"num_object_copies":99984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25023,"num_write_kb":199974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25050,"ondisk_log_size":25050,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":498552,"kb_used_data":307608,"kb_used_omap":2,"kb_used_meta":190909,"kb_avail":376988808,"statfs":{"total":386547056640,"available":386036539392,"internally_reserved":0,"allocated":314990592,"data_stored":314359000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":195490972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,11,19,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29917184,"num_objects":3652,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3652,"num_write_kb":29216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002508"},"pg_stats":[{"pgid":"1.f","version":"28'1495","reported_seq":"1533","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.075897+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:39.075897+0000","last_peered":"2021-01-15T18:40:39.075897+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:39.075897+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1495,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12247040,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":11960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":141}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1555","reported_seq":"1893","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.631679+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:38.631679+0000","last_peered":"2021-01-15T18:40:38.631679+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:38.631679+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1555,"ondisk_log_size":1555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12697766,"num_objects":1551,"num_object_clones":0,"num_object_copies":6204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":12412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1546","reported_seq":"1616","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:37.983962+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:37.983962+0000","last_peered":"2021-01-15T18:40:37.983962+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:37.983962+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1546,"ondisk_log_size":1546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12636243,"num_objects":1543,"num_object_clones":0,"num_object_copies":6172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":12346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1585","reported_seq":"1622","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.131922+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:39.131922+0000","last_peered":"2021-01-15T18:40:39.131922+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:39.131922+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1585,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12984320,"num_objects":1585,"num_object_clones":0,"num_object_copies":6340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1585,"num_write_kb":12680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1645","reported_seq":"1700","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.580566+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:38.580566+0000","last_peered":"2021-01-15T18:40:38.580566+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:38.580566+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1645,"ondisk_log_size":1645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13426854,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1643,"num_write_kb":13124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1475","reported_seq":"1546","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.096931+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:38.096931+0000","last_peered":"2021-01-15T18:40:38.096931+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:38.096931+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1475,"ondisk_log_size":1475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12050598,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":127}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1538","reported_seq":"1584","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.090407+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:39.090407+0000","last_peered":"2021-01-15T18:40:39.090407+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:39.090407+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1538,"ondisk_log_size":1538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12558502,"num_objects":1534,"num_object_clones":0,"num_object_copies":6136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1536,"num_write_kb":12276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1582","reported_seq":"1648","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.038994+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:38.038994+0000","last_peered":"2021-01-15T18:40:38.038994+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:38.038994+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1582,"ondisk_log_size":1582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12947539,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1581,"num_write_kb":12645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1610","reported_seq":"1644","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.083892+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:39.083892+0000","last_peered":"2021-01-15T18:40:39.083892+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:39.083892+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1610,"ondisk_log_size":1610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13176915,"num_objects":1609,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1609,"num_write_kb":12869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":173}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1607","reported_seq":"1658","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.138435+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:39.138435+0000","last_peered":"2021-01-15T18:40:39.138435+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:39.138435+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1607,"ondisk_log_size":1607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13111545,"num_objects":1602,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":12812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1533","reported_seq":"1583","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.006414+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:39.006414+0000","last_peered":"2021-01-15T18:40:39.006414+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:39.006414+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1533,"ondisk_log_size":1533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12525734,"num_objects":1530,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1531,"num_write_kb":12239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1516","reported_seq":"1572","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.602275+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:38.602275+0000","last_peered":"2021-01-15T18:40:38.602275+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:38.602275+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1516,"ondisk_log_size":1516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12378278,"num_objects":1512,"num_object_clones":0,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1543","reported_seq":"1601","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.599910+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:38.599910+0000","last_peered":"2021-01-15T18:40:38.599910+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:38.599910+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1543,"ondisk_log_size":1543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12583078,"num_objects":1537,"num_object_clones":0,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1541,"num_write_kb":12300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1599","reported_seq":"1656","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.601913+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:38.601913+0000","last_peered":"2021-01-15T18:40:38.601913+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:38.601913+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1599,"ondisk_log_size":1599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13050022,"num_objects":1594,"num_object_clones":0,"num_object_copies":6376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1597,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":152}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1694","reported_seq":"1746","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:39.131694+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:39.131694+0000","last_peered":"2021-01-15T18:40:39.131694+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:39.131694+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1694,"ondisk_log_size":1694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13795660,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1691,"num_write_kb":13491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1527","reported_seq":"1600","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:38.034347+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:38.034347+0000","last_peered":"2021-01-15T18:40:38.034347+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:38.034347+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1527,"ondisk_log_size":1527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12484774,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":12194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":204654868,"num_objects":24996,"num_object_clones":0,"num_object_copies":99984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25023,"num_write_kb":199974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317988864,"data_stored":317988864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25050,"ondisk_log_size":25050,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215118,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161536,"kb_used_data":99320,"kb_used_omap":0,"kb_used_meta":62207,"kb_avail":94210304,"statfs":{"total":96636764160,"available":96471351296,"internally_reserved":0,"allocated":101703680,"data_stored":101545782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":63700020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148516,"kb_used_data":98716,"kb_used_omap":0,"kb_used_meta":49792,"kb_avail":94223324,"statfs":{"total":96636764160,"available":96484683776,"internally_reserved":0,"allocated":101085184,"data_stored":100927286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50987008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149960,"kb_used_data":99776,"kb_used_omap":0,"kb_used_meta":50175,"kb_avail":94221880,"statfs":{"total":96636764160,"available":96483205120,"internally_reserved":0,"allocated":102170624,"data_stored":102012726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":51379252},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.387}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38540,"kb_used_data":9796,"kb_used_omap":0,"kb_used_meta":28735,"kb_avail":94333300,"statfs":{"total":96636764160,"available":96597299200,"internally_reserved":0,"allocated":10031104,"data_stored":9873206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":29424692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8010000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.165}]}]}]}} 2021-01-15T18:40:42.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:40:43.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 43 20 26717 26697 4.84896 2.82031 0.0417025 0.0322023 2021-01-15T18:40:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:43.526 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22528/99984 objects degraded (22.532%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":94,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24996,"data_bytes":204654868,"bytes_used":510517248,"bytes_avail":386036539392,"bytes_total":386547056640,"degraded_objects":22528,"degraded_total":99984,"degraded_ratio":0.22531605056809093,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":114659,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":2990968,"read_op_per_sec":0,"write_op_per_sec":365},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2021-01-15T18:40:14.363153+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:40:43.558 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:40:43.558 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:44.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 44 20 27097 27077 4.80623 2.96875 0.0478921 0.0324886 2021-01-15T18:40:44.509 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:44.564 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2021-01-15T18:40:44.374270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":221088020,"num_objects":27002,"num_object_clones":0,"num_object_copies":108008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27029,"num_write_kb":216022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27056,"ondisk_log_size":27056,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":542296,"kb_used_data":332104,"kb_used_omap":2,"kb_used_meta":210173,"kb_avail":376945064,"statfs":{"total":386547056640,"available":385991745536,"internally_reserved":0,"allocated":340074496,"data_stored":339442904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":215217308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,7,9,16,12,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46350336,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.895121"},"pg_stats":[{"pgid":"1.f","version":"28'1620","reported_seq":"1658","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.151394+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:44.151394+0000","last_peered":"2021-01-15T18:40:44.151394+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:44.151394+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13271040,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1620,"num_write_kb":12960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1679","reported_seq":"2219","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.610129+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:43.610129+0000","last_peered":"2021-01-15T18:40:43.610129+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:43.610129+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1679,"ondisk_log_size":1679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13713574,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1677,"num_write_kb":13404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1679","reported_seq":"1749","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.107048+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:43.107048+0000","last_peered":"2021-01-15T18:40:43.107048+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:43.107048+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1679,"ondisk_log_size":1679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13725779,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1678,"num_write_kb":13410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1706","reported_seq":"1744","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.084236+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:44.084236+0000","last_peered":"2021-01-15T18:40:44.084236+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:44.084236+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1706,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13975552,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1706,"num_write_kb":13648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1766","reported_seq":"1822","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.394244+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:43.394244+0000","last_peered":"2021-01-15T18:40:43.394244+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:43.394244+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1766,"ondisk_log_size":1766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14418086,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1764,"num_write_kb":14092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1601","reported_seq":"1671","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.078671+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:43.078671+0000","last_peered":"2021-01-15T18:40:43.078671+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:43.078671+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1601,"ondisk_log_size":1601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13082790,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1599,"num_write_kb":12783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":253}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1674","reported_seq":"1719","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.058659+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:44.058659+0000","last_peered":"2021-01-15T18:40:44.058659+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:44.058659+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1674,"ondisk_log_size":1674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13672614,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":13364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1716","reported_seq":"1782","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.035004+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:43.035004+0000","last_peered":"2021-01-15T18:40:43.035004+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:43.035004+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1716,"ondisk_log_size":1716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14045267,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1715,"num_write_kb":13717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1736","reported_seq":"1770","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.989128+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:43.989128+0000","last_peered":"2021-01-15T18:40:43.989128+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:43.989128+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1736,"ondisk_log_size":1736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14209107,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1735,"num_write_kb":13877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1744","reported_seq":"1796","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.111551+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:44.111551+0000","last_peered":"2021-01-15T18:40:44.111551+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:44.111551+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1744,"ondisk_log_size":1744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14233849,"num_objects":1739,"num_object_clones":0,"num_object_copies":6956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1741,"num_write_kb":13908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1645","reported_seq":"1695","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.053708+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:44.053708+0000","last_peered":"2021-01-15T18:40:44.053708+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:44.053708+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1645,"ondisk_log_size":1645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1643,"num_write_kb":13135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1639","reported_seq":"1695","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.632576+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:43.632576+0000","last_peered":"2021-01-15T18:40:43.632576+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:43.632576+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1639,"ondisk_log_size":1639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13385894,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1637,"num_write_kb":13079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1668","reported_seq":"1726","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.521050+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:43.521050+0000","last_peered":"2021-01-15T18:40:43.521050+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:43.521050+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1668,"ondisk_log_size":1668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13607078,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":13300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":257}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1749","reported_seq":"1806","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.595421+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:43.595421+0000","last_peered":"2021-01-15T18:40:43.595421+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:43.595421+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1749,"ondisk_log_size":1749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14278822,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1747,"num_write_kb":13956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1804","reported_seq":"1856","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.090268+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:44.090268+0000","last_peered":"2021-01-15T18:40:44.090268+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:44.090268+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1804,"ondisk_log_size":1804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14696780,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":14371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1630","reported_seq":"1703","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.133129+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:43.133129+0000","last_peered":"2021-01-15T18:40:43.133129+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:43.133129+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1630,"ondisk_log_size":1630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13328550,"num_objects":1628,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1628,"num_write_kb":13018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":221088020,"num_objects":27002,"num_object_clones":0,"num_object_copies":108008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27029,"num_write_kb":216022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":342773760,"data_stored":342773760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27056,"ondisk_log_size":27056,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165264,"kb_used_data":107528,"kb_used_omap":0,"kb_used_meta":57727,"kb_avail":94206576,"statfs":{"total":96636764160,"available":96467533824,"internally_reserved":0,"allocated":110108672,"data_stored":109950774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":59112500},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,5,2,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.617}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3849999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":168580,"kb_used_data":106876,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94203260,"statfs":{"total":96636764160,"available":96464138240,"internally_reserved":0,"allocated":109441024,"data_stored":109283126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0519999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":169912,"kb_used_data":107904,"kb_used_omap":0,"kb_used_meta":62015,"kb_avail":94201928,"statfs":{"total":96636764160,"available":96462774272,"internally_reserved":0,"allocated":110493696,"data_stored":110335798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":63503412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.984999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38540,"kb_used_data":9796,"kb_used_omap":0,"kb_used_meta":28735,"kb_avail":94333300,"statfs":{"total":96636764160,"available":96597299200,"internally_reserved":0,"allocated":10031104,"data_stored":9873206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":29424692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.702}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8010000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.165}]}]}]}} 2021-01-15T18:40:45.018 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 45 20 27492 27472 4.76801 3.08594 0.0640104 0.032744 2021-01-15T18:40:46.019 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 46 20 27857 27837 4.72622 2.85156 0.0360495 0.033044 2021-01-15T18:40:47.019 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 47 20 28248 28228 4.69066 3.05469 0.0472821 0.0332855 2021-01-15T18:40:47.567 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:48.020 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 48 20 28607 28587 4.65135 2.80469 0.0411261 0.0335505 2021-01-15T18:40:48.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:48.633 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:48.656 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2021-01-15T18:40:48.375257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":225241364,"num_objects":27509,"num_object_clones":0,"num_object_copies":110036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27536,"num_write_kb":220078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27563,"ondisk_log_size":27563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":563916,"kb_used_data":349756,"kb_used_omap":2,"kb_used_meta":214141,"kb_avail":376923444,"statfs":{"total":386547056640,"available":385969606656,"internally_reserved":0,"allocated":358150144,"data_stored":357518552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":219280540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,8,12,20,20,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36610048,"num_objects":4469,"num_object_clones":0,"num_object_copies":17876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":35752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"1.f","version":"28'1620","reported_seq":"1658","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.151394+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:44.151394+0000","last_peered":"2021-01-15T18:40:44.151394+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:44.151394+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13271040,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1620,"num_write_kb":12960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1679","reported_seq":"2219","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.610129+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:43.610129+0000","last_peered":"2021-01-15T18:40:43.610129+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:43.610129+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1679,"ondisk_log_size":1679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13713574,"num_objects":1675,"num_object_clones":0,"num_object_copies":6700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1677,"num_write_kb":13404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1808","reported_seq":"1878","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.100482+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:48.100482+0000","last_peered":"2021-01-15T18:40:48.100482+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:48.100482+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1808,"ondisk_log_size":1808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14782547,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1807,"num_write_kb":14442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1706","reported_seq":"1744","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.084236+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:44.084236+0000","last_peered":"2021-01-15T18:40:44.084236+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:44.084236+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1706,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13975552,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1706,"num_write_kb":13648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1766","reported_seq":"1822","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.394244+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:43.394244+0000","last_peered":"2021-01-15T18:40:43.394244+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:43.394244+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1766,"ondisk_log_size":1766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14418086,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1764,"num_write_kb":14092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1719","reported_seq":"1790","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.112812+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:48.112812+0000","last_peered":"2021-01-15T18:40:48.112812+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:48.112812+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1719,"ondisk_log_size":1719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14049446,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1717,"num_write_kb":13727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":371}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1674","reported_seq":"1719","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.058659+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:44.058659+0000","last_peered":"2021-01-15T18:40:44.058659+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:44.058659+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1674,"ondisk_log_size":1674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13672614,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":13364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1847","reported_seq":"1913","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.107328+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:48.107328+0000","last_peered":"2021-01-15T18:40:48.107328+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:48.107328+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1847,"ondisk_log_size":1847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15118419,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":14765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1736","reported_seq":"1770","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.989128+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:43.989128+0000","last_peered":"2021-01-15T18:40:43.989128+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:43.989128+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1736,"ondisk_log_size":1736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14209107,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1735,"num_write_kb":13877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1744","reported_seq":"1796","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.111551+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:44.111551+0000","last_peered":"2021-01-15T18:40:44.111551+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:44.111551+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1744,"ondisk_log_size":1744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14233849,"num_objects":1739,"num_object_clones":0,"num_object_copies":6956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1741,"num_write_kb":13908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1645","reported_seq":"1695","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.053708+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:44.053708+0000","last_peered":"2021-01-15T18:40:44.053708+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:44.053708+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1645,"ondisk_log_size":1645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1643,"num_write_kb":13135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1639","reported_seq":"1695","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.632576+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:43.632576+0000","last_peered":"2021-01-15T18:40:43.632576+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:43.632576+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1639,"ondisk_log_size":1639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13385894,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1637,"num_write_kb":13079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1668","reported_seq":"1726","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.521050+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:43.521050+0000","last_peered":"2021-01-15T18:40:43.521050+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:43.521050+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1668,"ondisk_log_size":1668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13607078,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1666,"num_write_kb":13300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":257}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1749","reported_seq":"1806","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:43.595421+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:43.595421+0000","last_peered":"2021-01-15T18:40:43.595421+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:43.595421+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1749,"ondisk_log_size":1749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14278822,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1747,"num_write_kb":13956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1804","reported_seq":"1856","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:44.090268+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:44.090268+0000","last_peered":"2021-01-15T18:40:44.090268+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:44.090268+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1804,"ondisk_log_size":1804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14696780,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":14371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1759","reported_seq":"1832","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.055621+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:48.055621+0000","last_peered":"2021-01-15T18:40:48.055621+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:48.055621+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1759,"ondisk_log_size":1759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14385318,"num_objects":1757,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":14050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":375}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":225241364,"num_objects":27509,"num_object_clones":0,"num_object_copies":110036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27536,"num_write_kb":220078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":359043072,"data_stored":359043072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27563,"ondisk_log_size":27563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215119,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165264,"kb_used_data":107528,"kb_used_omap":0,"kb_used_meta":57727,"kb_avail":94206576,"statfs":{"total":96636764160,"available":96467533824,"internally_reserved":0,"allocated":110108672,"data_stored":109950774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":59112500},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,5,2,3,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.617}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3849999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171392,"kb_used_data":114424,"kb_used_omap":0,"kb_used_meta":56960,"kb_avail":94200448,"statfs":{"total":96636764160,"available":96461258752,"internally_reserved":0,"allocated":117170176,"data_stored":117012278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58327040},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,14,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":22.417999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.931}]}]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":169912,"kb_used_data":107904,"kb_used_omap":0,"kb_used_meta":62015,"kb_avail":94201928,"statfs":{"total":96636764160,"available":96462774272,"internally_reserved":0,"allocated":110493696,"data_stored":110335798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":63503412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.984999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57348,"kb_used_data":19900,"kb_used_omap":0,"kb_used_meta":37439,"kb_avail":94314492,"statfs":{"total":96636764160,"available":96578039808,"internally_reserved":0,"allocated":20377600,"data_stored":20219702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":38337588},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5869999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2080000000000002}]}]}]}} 2021-01-15T18:40:48.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:40:49.020 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 49 20 28986 28966 4.61685 2.96094 0.0698861 0.0338202 2021-01-15T18:40:49.779 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:49.779 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22427/110036 objects degraded (20.382%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":100,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27509,"data_bytes":225241364,"bytes_used":577449984,"bytes_avail":385969606656,"bytes_total":386547056640,"degraded_objects":22427,"degraded_total":110036,"degraded_ratio":0.20381511505325531,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":150150,"recovering_keys_per_sec":0,"num_objects_recovered":220,"num_bytes_recovered":1802240,"num_keys_recovered":0,"write_bytes_sec":3050109,"read_op_per_sec":0,"write_op_per_sec":372},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-01-15T18:40:44.374396+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:40:49.824 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:40:49.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:50.020 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 50 20 29352 29332 4.58165 2.85938 0.0355054 0.0340788 2021-01-15T18:40:50.922 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:51.189 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2021-01-15T18:40:50.375737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":236824852,"num_objects":28923,"num_object_clones":0,"num_object_copies":115692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28950,"num_write_kb":231390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":3072000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28977,"ondisk_log_size":28977,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":599908,"kb_used_data":371616,"kb_used_omap":2,"kb_used_meta":228349,"kb_avail":376887452,"statfs":{"total":386547056640,"available":385932750848,"internally_reserved":0,"allocated":380534784,"data_stored":379903192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":233829532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,15,20,22,11,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44285952,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":43248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":2899968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002867"},"pg_stats":[{"pgid":"1.f","version":"28'1740","reported_seq":"1778","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.215541+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:49.215541+0000","last_peered":"2021-01-15T18:40:49.215541+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:49.215541+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1740,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14254080,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1740,"num_write_kb":13920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1789","reported_seq":"2596","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.665783+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:48.665783+0000","last_peered":"2021-01-15T18:40:48.665783+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:48.665783+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1789,"ondisk_log_size":1789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14614694,"num_objects":1785,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1787,"num_write_kb":14284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":377,"num_bytes_recovered":3072000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":742}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1808","reported_seq":"1878","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.100482+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:48.100482+0000","last_peered":"2021-01-15T18:40:48.100482+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:48.100482+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1808,"ondisk_log_size":1808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14782547,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1807,"num_write_kb":14442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1834","reported_seq":"1872","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.176581+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:49.176581+0000","last_peered":"2021-01-15T18:40:49.176581+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:49.176581+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1834,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15024128,"num_objects":1834,"num_object_clones":0,"num_object_copies":7336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":14672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":402}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'1899","reported_seq":"1955","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.606632+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:48.606632+0000","last_peered":"2021-01-15T18:40:48.606632+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:48.606632+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1899,"ondisk_log_size":1899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15507622,"num_objects":1894,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":15156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1719","reported_seq":"1790","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.112812+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:48.112812+0000","last_peered":"2021-01-15T18:40:48.112812+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:48.112812+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1719,"ondisk_log_size":1719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14049446,"num_objects":1716,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1717,"num_write_kb":13727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":371}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1800","reported_seq":"1846","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.181945+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:49.181945+0000","last_peered":"2021-01-15T18:40:49.181945+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:49.181945+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1800,"ondisk_log_size":1800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14704806,"num_objects":1796,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1798,"num_write_kb":14372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1847","reported_seq":"1913","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.107328+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:48.107328+0000","last_peered":"2021-01-15T18:40:48.107328+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:48.107328+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1847,"ondisk_log_size":1847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15118419,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":14765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":394}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1837","reported_seq":"1871","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.217638+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:49.217638+0000","last_peered":"2021-01-15T18:40:49.217638+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:49.217638+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1837,"ondisk_log_size":1837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15036499,"num_objects":1836,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1836,"num_write_kb":14685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":400}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1867","reported_seq":"1918","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.203025+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:49.203025+0000","last_peered":"2021-01-15T18:40:49.203025+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:49.203025+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1867,"ondisk_log_size":1867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15241465,"num_objects":1862,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1864,"num_write_kb":14892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1750","reported_seq":"1798","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.175306+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:49.175306+0000","last_peered":"2021-01-15T18:40:49.175306+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:49.175306+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1750,"ondisk_log_size":1750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14303398,"num_objects":1747,"num_object_clones":0,"num_object_copies":6988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1748,"num_write_kb":13975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1762","reported_seq":"1818","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.385890+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:48.385890+0000","last_peered":"2021-01-15T18:40:48.385890+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:48.385890+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1762,"ondisk_log_size":1762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14393510,"num_objects":1758,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1780","reported_seq":"1838","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.634227+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:48.634227+0000","last_peered":"2021-01-15T18:40:48.634227+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:48.634227+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1780,"ondisk_log_size":1780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14524582,"num_objects":1774,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1778,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1870","reported_seq":"1927","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.658502+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:48.658502+0000","last_peered":"2021-01-15T18:40:48.658502+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:48.658502+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1870,"ondisk_log_size":1870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15270054,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'1916","reported_seq":"1968","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:49.100531+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:49.100531+0000","last_peered":"2021-01-15T18:40:49.100531+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:49.100531+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1916,"ondisk_log_size":1916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15614284,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":15267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1759","reported_seq":"1832","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:48.055621+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:48.055621+0000","last_peered":"2021-01-15T18:40:48.055621+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:48.055621+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1759,"ondisk_log_size":1759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14385318,"num_objects":1757,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":14050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":375}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":236824852,"num_objects":28923,"num_object_clones":0,"num_object_copies":115692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28950,"num_write_kb":231390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":3072000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383066112,"data_stored":383066112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28977,"ondisk_log_size":28977,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":184516,"kb_used_data":115156,"kb_used_omap":0,"kb_used_meta":69439,"kb_avail":94187324,"statfs":{"total":96636764160,"available":96447819776,"internally_reserved":0,"allocated":117919744,"data_stored":117761846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":71105588},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,0,10,5,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.565}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3860000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171392,"kb_used_data":114424,"kb_used_omap":0,"kb_used_meta":56960,"kb_avail":94200448,"statfs":{"total":96636764160,"available":96461258752,"internally_reserved":0,"allocated":117170176,"data_stored":117012278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58327040},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,14,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":22.417999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.931}]}]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189240,"kb_used_data":115508,"kb_used_omap":0,"kb_used_meta":73727,"kb_avail":94182600,"statfs":{"total":96636764160,"available":96442982400,"internally_reserved":0,"allocated":118280192,"data_stored":118122294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75496500},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,8,6,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.323}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":1.4179999999999999},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":3.8809999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7880000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54760,"kb_used_data":26528,"kb_used_omap":0,"kb_used_meta":28223,"kb_avail":94317080,"statfs":{"total":96636764160,"available":96580689920,"internally_reserved":0,"allocated":27164672,"data_stored":27006774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":28900404},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5869999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2080000000000002}]}]}]}} 2021-01-15T18:40:51.190 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:51.191 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 51 20 29731 29711 4.54987 2.96094 0.0426663 0.0343169 2021-01-15T18:40:52.020 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 52 20 30070 30050 4.51331 2.64844 0.0560921 0.0346006 2021-01-15T18:40:53.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 53 20 30372 30352 4.47266 2.35938 0.0434183 0.034914 2021-01-15T18:40:54.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 54 20 30755 30735 4.44525 2.99219 0.0184122 0.0350966 2021-01-15T18:40:54.192 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:55.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 55 20 31075 31055 4.40989 2.5 0.0679171 0.0354107 2021-01-15T18:40:55.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:55.316 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:55.390 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2021-01-15T18:40:54.376720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":251046164,"num_objects":30659,"num_object_clones":0,"num_object_copies":122636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30686,"num_write_kb":245278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30713,"ondisk_log_size":30713,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":615244,"kb_used_data":393392,"kb_used_omap":2,"kb_used_meta":221821,"kb_avail":376872116,"statfs":{"total":386547056640,"available":385917046784,"internally_reserved":0,"allocated":402833408,"data_stored":402201816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":227144860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,8,6,9,6,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46391296,"num_objects":5663,"num_object_clones":0,"num_object_copies":22652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":45304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":3211264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"1.f","version":"28'1837","reported_seq":"1875","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.183103+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:54.183103+0000","last_peered":"2021-01-15T18:40:54.183103+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:54.183103+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1837,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15048704,"num_objects":1837,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":14696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1891","reported_seq":"3013","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.670565+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:53.670565+0000","last_peered":"2021-01-15T18:40:53.670565+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:53.670565+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1891,"ondisk_log_size":1891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15450278,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1889,"num_write_kb":15100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1908","reported_seq":"1979","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.078580+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:53.078580+0000","last_peered":"2021-01-15T18:40:53.078580+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:53.078580+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1908,"ondisk_log_size":1908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15601747,"num_objects":1905,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1907,"num_write_kb":15242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":489}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1945","reported_seq":"1983","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.188654+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:54.188654+0000","last_peered":"2021-01-15T18:40:54.188654+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:54.188654+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1945,"ondisk_log_size":1945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":15560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2005","reported_seq":"2061","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.702375+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:53.702375+0000","last_peered":"2021-01-15T18:40:53.702375+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:53.702375+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16375974,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1826","reported_seq":"1897","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.084208+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:53.084208+0000","last_peered":"2021-01-15T18:40:53.084208+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:53.084208+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1826,"ondisk_log_size":1826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14925990,"num_objects":1823,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1824,"num_write_kb":14583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1912","reported_seq":"1958","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.229410+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:54.229410+0000","last_peered":"2021-01-15T18:40:54.229410+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:54.229410+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1912,"ondisk_log_size":1912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15622310,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1910,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1971","reported_seq":"2036","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.129351+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:53.129351+0000","last_peered":"2021-01-15T18:40:53.129351+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:53.129351+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1971,"ondisk_log_size":1971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16134227,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":15757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1945","reported_seq":"1979","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.169415+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:54.169415+0000","last_peered":"2021-01-15T18:40:54.169415+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:54.169415+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1945,"ondisk_log_size":1945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15921235,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":15549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1973","reported_seq":"2025","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.228039+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:54.228039+0000","last_peered":"2021-01-15T18:40:54.228039+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:54.228039+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1973,"ondisk_log_size":1973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16109817,"num_objects":1968,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":15740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1850","reported_seq":"1900","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.187844+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:54.187844+0000","last_peered":"2021-01-15T18:40:54.187844+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:54.187844+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1850,"ondisk_log_size":1850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15122598,"num_objects":1847,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1872","reported_seq":"1928","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.657264+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:53.657264+0000","last_peered":"2021-01-15T18:40:53.657264+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:53.657264+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1872,"ondisk_log_size":1872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15294630,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1870,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1886","reported_seq":"1944","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.627404+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:53.627404+0000","last_peered":"2021-01-15T18:40:53.627404+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:53.627404+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1886,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15392934,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1991","reported_seq":"2048","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.627325+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:53.627325+0000","last_peered":"2021-01-15T18:40:53.627325+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:53.627325+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1991,"ondisk_log_size":1991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16261286,"num_objects":1986,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":15892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2035","reported_seq":"2086","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.229355+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:54.229355+0000","last_peered":"2021-01-15T18:40:54.229355+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:54.229355+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2035,"ondisk_log_size":2035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16589132,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":16219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1866","reported_seq":"1939","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:52.835087+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:52.835087+0000","last_peered":"2021-01-15T18:40:52.835087+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:52.835087+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1866,"ondisk_log_size":1866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15261862,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1864,"num_write_kb":14906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":251046164,"num_objects":30659,"num_object_clones":0,"num_object_copies":122636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30686,"num_write_kb":245278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":404799488,"data_stored":404799488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30713,"ondisk_log_size":30713,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182856,"kb_used_data":122436,"kb_used_omap":0,"kb_used_meta":60415,"kb_avail":94188984,"statfs":{"total":96636764160,"available":96449519616,"internally_reserved":0,"allocated":125374464,"data_stored":125216566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":61865012},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,0,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":13.532999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":11.406000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0039999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":8.5079999999999991},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":5.944}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":190044,"kb_used_data":121620,"kb_used_omap":0,"kb_used_meta":68416,"kb_avail":94181796,"statfs":{"total":96636764160,"available":96442159104,"internally_reserved":0,"allocated":124538880,"data_stored":124380982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70057984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2969999999999997},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.95799999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187584,"kb_used_data":122808,"kb_used_omap":0,"kb_used_meta":64767,"kb_avail":94184256,"statfs":{"total":96636764160,"available":96444678144,"internally_reserved":0,"allocated":125755392,"data_stored":125597494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":66321460},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":3.9390000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":22.765000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.55}]}]},{"osd":1,"up_from":26,"seq":111669149701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54760,"kb_used_data":26528,"kb_used_omap":0,"kb_used_meta":28223,"kb_avail":94317080,"statfs":{"total":96636764160,"available":96580689920,"internally_reserved":0,"allocated":27164672,"data_stored":27006774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":28900404},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5869999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2080000000000002}]}]}]}} 2021-01-15T18:40:55.392 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:40:56.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 56 20 31426 31406 4.38011 2.74219 0.0507826 0.0356531 2021-01-15T18:40:56.586 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:56.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22136/122636 objects degraded (18.050%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":107,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30659,"data_bytes":251046164,"bytes_used":645865472,"bytes_avail":385901191168,"bytes_total":386547056640,"degraded_objects":22136,"degraded_total":122636,"degraded_ratio":0.18050164715091813,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":198591,"recovering_keys_per_sec":0,"num_objects_recovered":291,"num_bytes_recovered":2383872,"num_keys_recovered":0,"write_bytes_sec":2495705,"read_op_per_sec":0,"write_op_per_sec":304},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:40:56.690 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:40:56.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:40:57.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 57 20 31808 31788 4.35562 2.98438 0.0395714 0.035859 2021-01-15T18:40:58.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 58 20 32179 32159 4.3305 2.89844 0.0566759 0.0360622 2021-01-15T18:40:58.034 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:40:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:40:58.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2021-01-15T18:40:56.378148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":251046164,"num_objects":30659,"num_object_clones":0,"num_object_copies":122636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30686,"num_write_kb":245278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30713,"ondisk_log_size":30713,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":630728,"kb_used_data":401196,"kb_used_omap":2,"kb_used_meta":229501,"kb_avail":376856632,"statfs":{"total":386547056640,"available":385901191168,"internally_reserved":0,"allocated":410824704,"data_stored":410193112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":235009180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,8,4,16,4,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29958144,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3657,"num_write_kb":29256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":2383872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"1.f","version":"28'1837","reported_seq":"1875","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.183103+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:54.183103+0000","last_peered":"2021-01-15T18:40:54.183103+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:54.183103+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1837,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15048704,"num_objects":1837,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":14696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1891","reported_seq":"3013","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.670565+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:53.670565+0000","last_peered":"2021-01-15T18:40:53.670565+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:53.670565+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1891,"ondisk_log_size":1891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15450278,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1889,"num_write_kb":15100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'1908","reported_seq":"1979","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.078580+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:53.078580+0000","last_peered":"2021-01-15T18:40:53.078580+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:53.078580+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1908,"ondisk_log_size":1908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15601747,"num_objects":1905,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1907,"num_write_kb":15242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":489}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'1945","reported_seq":"1983","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.188654+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:54.188654+0000","last_peered":"2021-01-15T18:40:54.188654+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:54.188654+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1945,"ondisk_log_size":1945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":15560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2005","reported_seq":"2061","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.702375+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:53.702375+0000","last_peered":"2021-01-15T18:40:53.702375+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:53.702375+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16375974,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":499}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1826","reported_seq":"1897","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.084208+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:53.084208+0000","last_peered":"2021-01-15T18:40:53.084208+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:53.084208+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1826,"ondisk_log_size":1826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14925990,"num_objects":1823,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1824,"num_write_kb":14583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'1912","reported_seq":"1958","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.229410+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:54.229410+0000","last_peered":"2021-01-15T18:40:54.229410+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:54.229410+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1912,"ondisk_log_size":1912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15622310,"num_objects":1908,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1910,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'1971","reported_seq":"2036","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.129351+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:53.129351+0000","last_peered":"2021-01-15T18:40:53.129351+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:53.129351+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1971,"ondisk_log_size":1971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16134227,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":15757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":518}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'1945","reported_seq":"1979","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.169415+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:54.169415+0000","last_peered":"2021-01-15T18:40:54.169415+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:54.169415+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1945,"ondisk_log_size":1945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15921235,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":15549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'1973","reported_seq":"2025","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.228039+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:54.228039+0000","last_peered":"2021-01-15T18:40:54.228039+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:54.228039+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1973,"ondisk_log_size":1973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16109817,"num_objects":1968,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":15740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":538}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1850","reported_seq":"1900","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.187844+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:54.187844+0000","last_peered":"2021-01-15T18:40:54.187844+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:54.187844+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1850,"ondisk_log_size":1850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15122598,"num_objects":1847,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":14775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1872","reported_seq":"1928","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.657264+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:53.657264+0000","last_peered":"2021-01-15T18:40:53.657264+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:53.657264+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1872,"ondisk_log_size":1872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15294630,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1870,"num_write_kb":14943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'1886","reported_seq":"1944","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.627404+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:53.627404+0000","last_peered":"2021-01-15T18:40:53.627404+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:53.627404+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1886,"ondisk_log_size":1886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15392934,"num_objects":1880,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'1991","reported_seq":"2048","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:53.627325+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:53.627325+0000","last_peered":"2021-01-15T18:40:53.627325+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:53.627325+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1991,"ondisk_log_size":1991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16261286,"num_objects":1986,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":15892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2035","reported_seq":"2086","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:54.229355+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:54.229355+0000","last_peered":"2021-01-15T18:40:54.229355+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:54.229355+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2035,"ondisk_log_size":2035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16589132,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":16219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'1866","reported_seq":"1939","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:52.835087+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:52.835087+0000","last_peered":"2021-01-15T18:40:52.835087+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:52.835087+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1866,"ondisk_log_size":1866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15261862,"num_objects":1864,"num_object_clones":0,"num_object_copies":7456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1864,"num_write_kb":14906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":251046164,"num_objects":30659,"num_object_clones":0,"num_object_copies":122636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30686,"num_write_kb":245278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":4358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":412581888,"data_stored":412581888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30713,"ondisk_log_size":30713,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215121,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182856,"kb_used_data":122436,"kb_used_omap":0,"kb_used_meta":60415,"kb_avail":94188984,"statfs":{"total":96636764160,"available":96449519616,"internally_reserved":0,"allocated":125374464,"data_stored":125216566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":61865012},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,0,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":13.532999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":11.406000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0039999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":8.5079999999999991},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":5.944}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":190044,"kb_used_data":121620,"kb_used_omap":0,"kb_used_meta":68416,"kb_avail":94181796,"statfs":{"total":96636764160,"available":96442159104,"internally_reserved":0,"allocated":124538880,"data_stored":124380982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70057984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2969999999999997},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.95799999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187584,"kb_used_data":122808,"kb_used_omap":0,"kb_used_meta":64767,"kb_avail":94184256,"statfs":{"total":96636764160,"available":96444678144,"internally_reserved":0,"allocated":125755392,"data_stored":125597494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":66321460},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":3.9390000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":22.765000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.55}]}]},{"osd":1,"up_from":26,"seq":111669149702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70244,"kb_used_data":34332,"kb_used_omap":0,"kb_used_meta":35903,"kb_avail":94301596,"statfs":{"total":96636764160,"available":96564834304,"internally_reserved":0,"allocated":35155968,"data_stored":34998070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":36764724},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.403}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0419999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.029999999999999}]}]}]}} 2021-01-15T18:40:59.021 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 59 20 32571 32551 4.30901 3.0625 0.0386247 0.0362218 2021-01-15T18:41:00.022 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:41:00.021848+0000 min lat: 0.00287162 max lat: 2.35257 avg lat: 0.036414 2021-01-15T18:41:00.022 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:41:00.022 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 60 20 32960 32940 4.28784 3.03906 0.0337433 0.036414 2021-01-15T18:41:01.023 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 61 20 33292 33272 4.26007 2.59375 0.0541651 0.0366587 2021-01-15T18:41:01.097 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:02.024 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 62 20 33642 33622 4.23533 2.73438 0.0592835 0.0368716 2021-01-15T18:41:02.135 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:02.153 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:02.186 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2021-01-15T18:41:00.379087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266234132,"num_objects":32513,"num_object_clones":0,"num_object_copies":130052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32540,"num_write_kb":260110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32567,"ondisk_log_size":32567,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":687660,"kb_used_data":429196,"kb_used_omap":2,"kb_used_meta":258429,"kb_avail":376799700,"statfs":{"total":386547056640,"available":385842892800,"internally_reserved":0,"allocated":439496704,"data_stored":438865112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":264631452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,11,12,14,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40992768,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003830"},"pg_stats":[{"pgid":"1.f","version":"28'1943","reported_seq":"1981","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.234425+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:59.234425+0000","last_peered":"2021-01-15T18:40:59.234425+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:59.234425+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1943,"ondisk_log_size":1943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15917056,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1943,"num_write_kb":15544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2005","reported_seq":"3421","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.780413+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:58.780413+0000","last_peered":"2021-01-15T18:40:58.780413+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:58.780413+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16384166,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2014","reported_seq":"2085","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.075265+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:58.075265+0000","last_peered":"2021-01-15T18:40:58.075265+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:58.075265+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2014,"ondisk_log_size":2014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16470099,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":16090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2059","reported_seq":"2097","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.255521+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:59.255521+0000","last_peered":"2021-01-15T18:40:59.255521+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:59.255521+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2059,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16867328,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2107","reported_seq":"2161","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.757703+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:58.757703+0000","last_peered":"2021-01-15T18:40:58.757703+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:58.757703+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2107,"ondisk_log_size":2107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17211558,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2105,"num_write_kb":16820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1944","reported_seq":"2015","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.149799+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:58.149799+0000","last_peered":"2021-01-15T18:40:58.149799+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:58.149799+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1944,"ondisk_log_size":1944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15892646,"num_objects":1941,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1942,"num_write_kb":15527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2016","reported_seq":"2062","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.085587+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:59.085587+0000","last_peered":"2021-01-15T18:40:59.085587+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:59.085587+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2016,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16474278,"num_objects":2012,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":16100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2087","reported_seq":"2153","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:57.990778+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:57.990778+0000","last_peered":"2021-01-15T18:40:57.990778+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:57.990778+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2087,"ondisk_log_size":2087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17084499,"num_objects":2086,"num_object_clones":0,"num_object_copies":8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2066","reported_seq":"2099","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.258475+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:59.258475+0000","last_peered":"2021-01-15T18:40:59.258475+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:59.258475+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2066,"ondisk_log_size":2066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16912467,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2085","reported_seq":"2137","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.171232+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:59.171232+0000","last_peered":"2021-01-15T18:40:59.171232+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:59.171232+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2085,"ondisk_log_size":2085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17027321,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2082,"num_write_kb":16636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1966","reported_seq":"2014","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.207793+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:59.207793+0000","last_peered":"2021-01-15T18:40:59.207793+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:59.207793+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1966,"ondisk_log_size":1966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16072870,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1964,"num_write_kb":15703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1979","reported_seq":"2035","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.775047+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:58.775047+0000","last_peered":"2021-01-15T18:40:58.775047+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:58.775047+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1979,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16171174,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2006","reported_seq":"2064","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.756322+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:58.756322+0000","last_peered":"2021-01-15T18:40:58.756322+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:58.756322+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2006,"ondisk_log_size":2006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16375974,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":16004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":595}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2119","reported_seq":"2176","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.739213+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:58.739213+0000","last_peered":"2021-01-15T18:40:58.739213+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:58.739213+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2119,"ondisk_log_size":2119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17309862,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2117,"num_write_kb":16916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2166","reported_seq":"2218","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.250882+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:59.250882+0000","last_peered":"2021-01-15T18:40:59.250882+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:59.250882+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2166,"ondisk_log_size":2166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17662284,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":17267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2005","reported_seq":"2078","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.102711+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:58.102711+0000","last_peered":"2021-01-15T18:40:58.102711+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:58.102711+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16400550,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266234132,"num_objects":32513,"num_object_clones":0,"num_object_copies":130052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32540,"num_write_kb":260110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":443269120,"data_stored":443269120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32567,"ondisk_log_size":32567,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197552,"kb_used_data":129768,"kb_used_omap":0,"kb_used_meta":67775,"kb_avail":94174288,"statfs":{"total":96636764160,"available":96434470912,"internally_reserved":0,"allocated":132882432,"data_stored":132724534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":69401652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":1.3959999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":1.974}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":4.9359999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":4.4740000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204444,"kb_used_data":128724,"kb_used_omap":0,"kb_used_meta":75712,"kb_avail":94167396,"statfs":{"total":96636764160,"available":96427413504,"internally_reserved":0,"allocated":131813376,"data_stored":131655478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77529088},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":15.912000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.2789999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":3.7869999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":1.056}]}]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200000,"kb_used_data":128568,"kb_used_omap":0,"kb_used_meta":71423,"kb_avail":94171840,"statfs":{"total":96636764160,"available":96431964160,"internally_reserved":0,"allocated":131653632,"data_stored":131495734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73137204},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":2.3940000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":3.9870000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.80300000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85664,"kb_used_data":42136,"kb_used_omap":0,"kb_used_meta":43519,"kb_avail":94286176,"statfs":{"total":96636764160,"available":96549044224,"internally_reserved":0,"allocated":43147264,"data_stored":42989366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":44563508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2779999999999996}]}]}]}} 2021-01-15T18:41:02.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:03.023 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 63 20 33964 33944 4.20804 2.51562 0.0490796 0.0371108 2021-01-15T18:41:03.295 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:03.296 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21990/130052 objects degraded (16.909%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":113,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32513,"data_bytes":266234132,"bytes_used":704163840,"bytes_avail":385842892800,"bytes_total":386547056640,"degraded_objects":21990,"degraded_total":130052,"degraded_ratio":0.16908621166917848,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":206783,"recovering_keys_per_sec":0,"num_objects_recovered":305,"num_bytes_recovered":2482176,"num_keys_recovered":0,"write_bytes_sec":2450014,"read_op_per_sec":0,"write_op_per_sec":299},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:03.360 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:03.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:04.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 64 20 34323 34303 4.18602 2.80469 0.10094 0.0373002 2021-01-15T18:41:04.334 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:04.396 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2021-01-15T18:41:02.379455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266234132,"num_objects":32513,"num_object_clones":0,"num_object_copies":130052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32540,"num_write_kb":260110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32567,"ondisk_log_size":32567,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":687660,"kb_used_data":429196,"kb_used_omap":2,"kb_used_meta":258429,"kb_avail":376799700,"statfs":{"total":386547056640,"available":385842892800,"internally_reserved":0,"allocated":439496704,"data_stored":438865112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":264631452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,11,12,14,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29409280,"num_objects":3590,"num_object_clones":0,"num_object_copies":14360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":2482176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"1.f","version":"28'1943","reported_seq":"1981","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.234425+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:40:59.234425+0000","last_peered":"2021-01-15T18:40:59.234425+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:40:59.234425+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1943,"ondisk_log_size":1943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15917056,"num_objects":1943,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1943,"num_write_kb":15544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2005","reported_seq":"3421","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.780413+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:40:58.780413+0000","last_peered":"2021-01-15T18:40:58.780413+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:40:58.780413+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16384166,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2014","reported_seq":"2085","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.075265+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:40:58.075265+0000","last_peered":"2021-01-15T18:40:58.075265+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:40:58.075265+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2014,"ondisk_log_size":2014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16470099,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":16090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2059","reported_seq":"2097","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.255521+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:40:59.255521+0000","last_peered":"2021-01-15T18:40:59.255521+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:40:59.255521+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2059,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16867328,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":16472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2107","reported_seq":"2161","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.757703+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:40:58.757703+0000","last_peered":"2021-01-15T18:40:58.757703+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:40:58.757703+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2107,"ondisk_log_size":2107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17211558,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2105,"num_write_kb":16820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'1944","reported_seq":"2015","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.149799+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:40:58.149799+0000","last_peered":"2021-01-15T18:40:58.149799+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:40:58.149799+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1944,"ondisk_log_size":1944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15892646,"num_objects":1941,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1942,"num_write_kb":15527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2016","reported_seq":"2062","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.085587+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:40:59.085587+0000","last_peered":"2021-01-15T18:40:59.085587+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:40:59.085587+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2016,"ondisk_log_size":2016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16474278,"num_objects":2012,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":16100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":620}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2087","reported_seq":"2153","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:57.990778+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:40:57.990778+0000","last_peered":"2021-01-15T18:40:57.990778+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:40:57.990778+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2087,"ondisk_log_size":2087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17084499,"num_objects":2086,"num_object_clones":0,"num_object_copies":8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":16685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2066","reported_seq":"2099","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.258475+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:40:59.258475+0000","last_peered":"2021-01-15T18:40:59.258475+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:40:59.258475+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2066,"ondisk_log_size":2066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16912467,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2085","reported_seq":"2137","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.171232+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:40:59.171232+0000","last_peered":"2021-01-15T18:40:59.171232+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:40:59.171232+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2085,"ondisk_log_size":2085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17027321,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2082,"num_write_kb":16636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'1966","reported_seq":"2014","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.207793+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:40:59.207793+0000","last_peered":"2021-01-15T18:40:59.207793+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:40:59.207793+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1966,"ondisk_log_size":1966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16072870,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1964,"num_write_kb":15703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'1979","reported_seq":"2035","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.775047+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:40:58.775047+0000","last_peered":"2021-01-15T18:40:58.775047+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:40:58.775047+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":1979,"ondisk_log_size":1979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16171174,"num_objects":1975,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":15799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2006","reported_seq":"2064","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.756322+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:40:58.756322+0000","last_peered":"2021-01-15T18:40:58.756322+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:40:58.756322+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2006,"ondisk_log_size":2006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16375974,"num_objects":2000,"num_object_clones":0,"num_object_copies":8000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":16004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":595}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2119","reported_seq":"2176","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.739213+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:40:58.739213+0000","last_peered":"2021-01-15T18:40:58.739213+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:40:58.739213+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2119,"ondisk_log_size":2119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17309862,"num_objects":2114,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2117,"num_write_kb":16916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":672}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2166","reported_seq":"2218","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:59.250882+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:40:59.250882+0000","last_peered":"2021-01-15T18:40:59.250882+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:40:59.250882+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2166,"ondisk_log_size":2166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17662284,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":17267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2005","reported_seq":"2078","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:40:58.102711+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:40:58.102711+0000","last_peered":"2021-01-15T18:40:58.102711+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:40:58.102711+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2005,"ondisk_log_size":2005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16400550,"num_objects":2003,"num_object_clones":0,"num_object_copies":8012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":16018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266234132,"num_objects":32513,"num_object_clones":0,"num_object_copies":130052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32540,"num_write_kb":260110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":443269120,"data_stored":443269120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32567,"ondisk_log_size":32567,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215122,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197552,"kb_used_data":129768,"kb_used_omap":0,"kb_used_meta":67775,"kb_avail":94174288,"statfs":{"total":96636764160,"available":96434470912,"internally_reserved":0,"allocated":132882432,"data_stored":132724534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":69401652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":1.3959999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":1.974}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":4.9359999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":4.4740000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204444,"kb_used_data":128724,"kb_used_omap":0,"kb_used_meta":75712,"kb_avail":94167396,"statfs":{"total":96636764160,"available":96427413504,"internally_reserved":0,"allocated":131813376,"data_stored":131655478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77529088},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":15.912000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.2789999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":3.7869999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":1.056}]}]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200000,"kb_used_data":128568,"kb_used_omap":0,"kb_used_meta":71423,"kb_avail":94171840,"statfs":{"total":96636764160,"available":96431964160,"internally_reserved":0,"allocated":131653632,"data_stored":131495734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73137204},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":2.3940000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":3.9870000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.80300000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85664,"kb_used_data":42136,"kb_used_omap":0,"kb_used_meta":43519,"kb_avail":94286176,"statfs":{"total":96636764160,"available":96549044224,"internally_reserved":0,"allocated":43147264,"data_stored":42989366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":44563508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2779999999999996}]}]}]}} 2021-01-15T18:41:05.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 65 20 34696 34676 4.16645 2.91406 0.0673296 0.0374777 2021-01-15T18:41:06.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 66 20 34982 34962 4.13718 2.23438 0.0625861 0.0377472 2021-01-15T18:41:07.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 67 20 35333 35313 4.11636 2.74219 0.0510102 0.037934 2021-01-15T18:41:07.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:08.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 68 20 35698 35678 4.09776 2.85156 0.0502355 0.0381091 2021-01-15T18:41:08.407 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:08.417 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:08.438 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2021-01-15T18:41:08.382379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":284100884,"num_objects":34694,"num_object_clones":0,"num_object_copies":138776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34721,"num_write_kb":277558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34748,"ondisk_log_size":34748,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":744288,"kb_used_data":465244,"kb_used_omap":2,"kb_used_meta":279037,"kb_avail":376743072,"statfs":{"total":386547056640,"available":385784905728,"internally_reserved":0,"allocated":476409856,"data_stored":475778264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":285734044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,9,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33054720,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":32280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":2572288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004231"},"pg_stats":[{"pgid":"1.f","version":"28'2047","reported_seq":"2085","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.252617+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:04.252617+0000","last_peered":"2021-01-15T18:41:04.252617+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:04.252617+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2047,"ondisk_log_size":2047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16769024,"num_objects":2047,"num_object_clones":0,"num_object_copies":8188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2047,"num_write_kb":16376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2103","reported_seq":"3855","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:03.792664+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:03.792664+0000","last_peered":"2021-01-15T18:41:03.792664+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:03.792664+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2103,"ondisk_log_size":2103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17186982,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":16796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":850,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2218","reported_seq":"2289","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.159355+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:08.159355+0000","last_peered":"2021-01-15T18:41:08.159355+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:08.159355+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2218,"ondisk_log_size":2218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18141267,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":17722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2187","reported_seq":"2225","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.196388+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:04.196388+0000","last_peered":"2021-01-15T18:41:04.196388+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:04.196388+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2187,"ondisk_log_size":2187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17915904,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":17496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2218","reported_seq":"2272","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:03.784476+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:03.784476+0000","last_peered":"2021-01-15T18:41:03.784476+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:03.784476+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2218,"ondisk_log_size":2218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18120870,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":17708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2178","reported_seq":"2249","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.096402+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:08.096402+0000","last_peered":"2021-01-15T18:41:08.096402+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:08.096402+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2178,"ondisk_log_size":2178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17809574,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2176,"num_write_kb":17399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2146","reported_seq":"2192","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.105858+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:04.105858+0000","last_peered":"2021-01-15T18:41:04.105858+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:04.105858+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2146,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17539238,"num_objects":2142,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2286","reported_seq":"2352","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.056245+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:08.056245+0000","last_peered":"2021-01-15T18:41:08.056245+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:08.056245+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2286,"ondisk_log_size":2286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18714707,"num_objects":2285,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2285,"num_write_kb":18277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2185","reported_seq":"2219","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.234410+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:04.234410+0000","last_peered":"2021-01-15T18:41:04.234410+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:04.234410+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2185,"ondisk_log_size":2185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17887315,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":17469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2192","reported_seq":"2244","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.251075+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:04.251075+0000","last_peered":"2021-01-15T18:41:04.251075+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:04.251075+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2192,"ondisk_log_size":2192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17903865,"num_objects":2187,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":17492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2081","reported_seq":"2131","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.252120+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:04.252120+0000","last_peered":"2021-01-15T18:41:04.252120+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:04.252120+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2081,"ondisk_log_size":2081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17014950,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":16623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2071","reported_seq":"2127","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:03.667437+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:03.667437+0000","last_peered":"2021-01-15T18:41:03.667437+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:03.667437+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2071,"ondisk_log_size":2071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16924838,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":16535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":696}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2108","reported_seq":"2166","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:03.668905+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:03.668905+0000","last_peered":"2021-01-15T18:41:03.668905+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:03.668905+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2108,"ondisk_log_size":2108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17211558,"num_objects":2102,"num_object_clones":0,"num_object_copies":8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":16820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2239","reported_seq":"2295","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:03.781560+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:03.781560+0000","last_peered":"2021-01-15T18:41:03.781560+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:03.781560+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2239,"ondisk_log_size":2239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18292902,"num_objects":2234,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2237,"num_write_kb":17876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2271","reported_seq":"2322","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:04.233821+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:04.233821+0000","last_peered":"2021-01-15T18:41:04.233821+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:04.233821+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2271,"ondisk_log_size":2271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18522444,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":18107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2218","reported_seq":"2291","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.101662+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:08.101662+0000","last_peered":"2021-01-15T18:41:08.101662+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:08.101662+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2218,"ondisk_log_size":2218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18145446,"num_objects":2216,"num_object_clones":0,"num_object_copies":8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":17722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":284100884,"num_objects":34694,"num_object_clones":0,"num_object_copies":138776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34721,"num_write_kb":277558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":6930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479956992,"data_stored":479956992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34748,"ondisk_log_size":34748,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215123,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":216044,"kb_used_data":136808,"kb_used_omap":0,"kb_used_meta":79231,"kb_avail":94155796,"statfs":{"total":96636764160,"available":96415535104,"internally_reserved":0,"allocated":140091392,"data_stored":139933494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81132596},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":2.4239999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":2.0609999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":8.0709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220828,"kb_used_data":142824,"kb_used_omap":0,"kb_used_meta":78016,"kb_avail":94151012,"statfs":{"total":96636764160,"available":96410636288,"internally_reserved":0,"allocated":146251776,"data_stored":146093878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79888384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.423},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":3.7469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":1.319}]}]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":202280,"kb_used_data":135776,"kb_used_omap":0,"kb_used_meta":66495,"kb_avail":94169560,"statfs":{"total":96636764160,"available":96429629440,"internally_reserved":0,"allocated":139034624,"data_stored":138876726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":68090932},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.64400000000000002},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.878}]}]},{"osd":1,"up_from":26,"seq":111669149704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105136,"kb_used_data":49836,"kb_used_omap":0,"kb_used_meta":55295,"kb_avail":94266704,"statfs":{"total":96636764160,"available":96529104896,"internally_reserved":0,"allocated":51032064,"data_stored":50874166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":56622132},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2450000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9989999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2200000000000002}]}]}]}} 2021-01-15T18:41:08.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:09.026 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 69 20 36041 36021 4.07721 2.67969 0.0510825 0.0383019 2021-01-15T18:41:09.733 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:09.733 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21822/138776 objects degraded (15.725%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":120,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34694,"data_bytes":284100884,"bytes_used":762150912,"bytes_avail":385784905728,"bytes_total":386547056640,"degraded_objects":21822,"degraded_total":138776,"degraded_ratio":0.15724620971925982,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":214281,"recovering_keys_per_sec":0,"num_objects_recovered":316,"num_bytes_recovered":2572288,"num_keys_recovered":0,"write_bytes_sec":2753589,"read_op_per_sec":0,"write_op_per_sec":336},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:09.782 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:09.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:10.025 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 70 20 36375 36355 4.05624 2.60938 0.0431985 0.038489 2021-01-15T18:41:10.853 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:10.892 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:10.954 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2021-01-15T18:41:10.382960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":294799636,"num_objects":36000,"num_object_clones":0,"num_object_copies":144000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36027,"num_write_kb":288006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36054,"ondisk_log_size":36054,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":760832,"kb_used_data":487868,"kb_used_omap":2,"kb_used_meta":272957,"kb_avail":376726528,"statfs":{"total":386547056640,"available":385767964672,"internally_reserved":0,"allocated":499576832,"data_stored":498945240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":279508124},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,4,6,11,8,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39829504,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":3899392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004346"},"pg_stats":[{"pgid":"1.f","version":"28'2162","reported_seq":"2200","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.183618+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:09.183618+0000","last_peered":"2021-01-15T18:41:09.183618+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:09.183618+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2162,"ondisk_log_size":2162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17711104,"num_objects":2162,"num_object_clones":0,"num_object_copies":8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":17296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2224","reported_seq":"4302","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.759651+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:08.759651+0000","last_peered":"2021-01-15T18:41:08.759651+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:08.759651+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2224,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18178214,"num_objects":2220,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1810}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2218","reported_seq":"2289","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.159355+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:08.159355+0000","last_peered":"2021-01-15T18:41:08.159355+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:08.159355+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2218,"ondisk_log_size":2218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18141267,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":17722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2300","reported_seq":"2338","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.255363+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:09.255363+0000","last_peered":"2021-01-15T18:41:09.255363+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:09.255363+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2300,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18841600,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2329","reported_seq":"2385","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.715270+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:08.715270+0000","last_peered":"2021-01-15T18:41:08.715270+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:08.715270+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2329,"ondisk_log_size":2329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19030182,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2327,"num_write_kb":18596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":823}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2178","reported_seq":"2249","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.096402+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:08.096402+0000","last_peered":"2021-01-15T18:41:08.096402+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:08.096402+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2178,"ondisk_log_size":2178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17809574,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2176,"num_write_kb":17399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2240","reported_seq":"2285","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.285748+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:09.285748+0000","last_peered":"2021-01-15T18:41:09.285748+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:09.285748+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2240,"ondisk_log_size":2240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18309286,"num_objects":2236,"num_object_clones":0,"num_object_copies":8944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2286","reported_seq":"2352","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.056245+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:08.056245+0000","last_peered":"2021-01-15T18:41:08.056245+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:08.056245+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2286,"ondisk_log_size":2286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18714707,"num_objects":2285,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2285,"num_write_kb":18277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2281","reported_seq":"2315","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.225451+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:09.225451+0000","last_peered":"2021-01-15T18:41:09.225451+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:09.225451+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2281,"ondisk_log_size":2281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18673747,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":18237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":844}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2307","reported_seq":"2359","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.072258+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:09.072258+0000","last_peered":"2021-01-15T18:41:09.072258+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:09.072258+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2307,"ondisk_log_size":2307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18845945,"num_objects":2302,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2189","reported_seq":"2238","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.285763+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:09.285763+0000","last_peered":"2021-01-15T18:41:09.285763+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:09.285763+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2189,"ondisk_log_size":2189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17899686,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":17487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2186","reported_seq":"2242","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.768008+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:08.768008+0000","last_peered":"2021-01-15T18:41:08.768008+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:08.768008+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2186,"ondisk_log_size":2186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17866918,"num_objects":2182,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":17455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2211","reported_seq":"2268","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.763963+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:08.763963+0000","last_peered":"2021-01-15T18:41:08.763963+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:08.763963+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2211,"ondisk_log_size":2211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18055334,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2209,"num_write_kb":17644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2340","reported_seq":"2397","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.778824+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:08.778824+0000","last_peered":"2021-01-15T18:41:08.778824+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:08.778824+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2340,"ondisk_log_size":2340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19120294,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2385","reported_seq":"2436","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:09.246215+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:09.246215+0000","last_peered":"2021-01-15T18:41:09.246215+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:09.246215+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2385,"ondisk_log_size":2385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19456332,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2382,"num_write_kb":19019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2218","reported_seq":"2291","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:08.101662+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:08.101662+0000","last_peered":"2021-01-15T18:41:08.101662+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:08.101662+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2218,"ondisk_log_size":2218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18145446,"num_objects":2216,"num_object_clones":0,"num_object_copies":8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":17722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":294799636,"num_objects":36000,"num_object_clones":0,"num_object_copies":144000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36027,"num_write_kb":288006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501747712,"data_stored":501747712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36054,"ondisk_log_size":36054,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215124,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":213772,"kb_used_data":143692,"kb_used_omap":0,"kb_used_meta":70079,"kb_avail":94158068,"statfs":{"total":96636764160,"available":96417861632,"internally_reserved":0,"allocated":147140608,"data_stored":146982710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":71760948},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":2.911},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":4.1340000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2400000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":9.2119999999999997},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":3.0169999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220828,"kb_used_data":142824,"kb_used_omap":0,"kb_used_meta":78016,"kb_avail":94151012,"statfs":{"total":96636764160,"available":96410636288,"internally_reserved":0,"allocated":146251776,"data_stored":146093878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79888384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.423},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":3.7469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":1.319}]}]},{"osd":0,"up_from":22,"seq":94489280533,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":216392,"kb_used_data":142656,"kb_used_omap":0,"kb_used_meta":73727,"kb_avail":94155448,"statfs":{"total":96636764160,"available":96415178752,"internally_reserved":0,"allocated":146079744,"data_stored":145921846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75496500},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,0,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":27.77},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":7.194}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":3.3860000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":4.6029999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109840,"kb_used_data":58696,"kb_used_omap":0,"kb_used_meta":51135,"kb_avail":94262000,"statfs":{"total":96636764160,"available":96524288000,"internally_reserved":0,"allocated":60104704,"data_stored":59946806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":52362292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.234}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.745000000000001}]}]}]}} 2021-01-15T18:41:11.026 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 71 20 36731 36711 4.03829 2.78125 0.0424946 0.038667 2021-01-15T18:41:12.026 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 72 20 37113 37093 4.02365 2.98438 0.0388779 0.0388146 2021-01-15T18:41:13.029 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 73 20 37434 37414 4.00272 2.50781 0.20759 0.0390097 2021-01-15T18:41:13.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:14.029 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 74 20 37801 37781 3.98738 2.86719 0.0502905 0.0391704 2021-01-15T18:41:15.006 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:15.008 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:15.018 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2021-01-15T18:41:14.385897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309201172,"num_objects":37758,"num_object_clones":0,"num_object_copies":151032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37785,"num_write_kb":302070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37812,"ondisk_log_size":37812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":800160,"kb_used_data":509400,"kb_used_omap":2,"kb_used_meta":290813,"kb_avail":376687200,"statfs":{"total":386547056640,"available":385727692800,"internally_reserved":0,"allocated":521625600,"data_stored":520994008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":297792668},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,8,14,16,25,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":64,"apply_latency_ms":64,"commit_latency_ns":64000000,"apply_latency_ns":64000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42967040,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":483,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006442"},"pg_stats":[{"pgid":"1.f","version":"28'2252","reported_seq":"2290","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.232133+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:14.232133+0000","last_peered":"2021-01-15T18:41:14.232133+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:14.232133+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2252,"ondisk_log_size":2252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18448384,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":18016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2342","reported_seq":"4726","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.786303+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:13.786303+0000","last_peered":"2021-01-15T18:41:13.786303+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:13.786303+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2342,"ondisk_log_size":2342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19144870,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":18708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2329","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.119061+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:13.119061+0000","last_peered":"2021-01-15T18:41:13.119061+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:13.119061+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2329,"ondisk_log_size":2329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19050579,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2328,"num_write_kb":18610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2423","reported_seq":"2461","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.115004+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:14.115004+0000","last_peered":"2021-01-15T18:41:14.115004+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:14.115004+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2423,"ondisk_log_size":2423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19849216,"num_objects":2423,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2423,"num_write_kb":19384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2436","reported_seq":"2492","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.641558+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:13.641558+0000","last_peered":"2021-01-15T18:41:13.641558+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:13.641558+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2436,"ondisk_log_size":2436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19906726,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":19452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2303","reported_seq":"2373","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.192313+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:13.192313+0000","last_peered":"2021-01-15T18:41:13.192313+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:13.192313+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2303,"ondisk_log_size":2303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18833574,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":18399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2336","reported_seq":"2381","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.185552+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:14.185552+0000","last_peered":"2021-01-15T18:41:14.185552+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:14.185552+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2336,"ondisk_log_size":2336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19095718,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2334,"num_write_kb":18660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2386","reported_seq":"2452","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.123327+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:13.123327+0000","last_peered":"2021-01-15T18:41:13.123327+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:13.123327+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2386,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19533907,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2406","reported_seq":"2438","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.258191+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:14.258191+0000","last_peered":"2021-01-15T18:41:14.258191+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:14.258191+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19697747,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2405,"num_write_kb":19237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2434","reported_seq":"2486","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.264460+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:14.264460+0000","last_peered":"2021-01-15T18:41:14.264460+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:14.264460+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2434,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19886329,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2280","reported_seq":"2330","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.221305+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:14.221305+0000","last_peered":"2021-01-15T18:41:14.221305+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:14.221305+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2280,"ondisk_log_size":2280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18645158,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2278,"num_write_kb":18215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2288","reported_seq":"2344","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.580993+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:13.580993+0000","last_peered":"2021-01-15T18:41:13.580993+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:13.580993+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2288,"ondisk_log_size":2288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18702502,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2286,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":913}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2314","reported_seq":"2371","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.773175+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:13.773175+0000","last_peered":"2021-01-15T18:41:13.773175+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:13.773175+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2314,"ondisk_log_size":2314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18899110,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2451","reported_seq":"2508","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.793840+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:13.793840+0000","last_peered":"2021-01-15T18:41:13.793840+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:13.793840+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2451,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2505","reported_seq":"2556","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.256874+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:14.256874+0000","last_peered":"2021-01-15T18:41:14.256874+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:14.256874+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2505,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20439372,"num_objects":2497,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":19979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2327","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.080733+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:13.080733+0000","last_peered":"2021-01-15T18:41:13.080733+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:13.080733+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2327,"ondisk_log_size":2327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19038374,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2325,"num_write_kb":18594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309201172,"num_objects":37758,"num_object_clones":0,"num_object_copies":151032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37785,"num_write_kb":302070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":523247616,"data_stored":523247616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37812,"ondisk_log_size":37812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232208,"kb_used_data":150812,"kb_used_omap":0,"kb_used_meta":81471,"kb_avail":94139632,"statfs":{"total":96636764160,"available":96398983168,"internally_reserved":0,"allocated":154431488,"data_stored":154273590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":83426356},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":12.91},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":1.4730000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":223016,"kb_used_data":149988,"kb_used_omap":0,"kb_used_meta":73024,"kb_avail":94148824,"statfs":{"total":96636764160,"available":96408395776,"internally_reserved":0,"allocated":153587712,"data_stored":153429814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74776576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5840000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.7249999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2469999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":2.2570000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235096,"kb_used_data":149904,"kb_used_omap":0,"kb_used_meta":85183,"kb_avail":94136744,"statfs":{"total":96636764160,"available":96396025856,"internally_reserved":0,"allocated":153501696,"data_stored":153343798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87227444},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4159999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":7.202},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":14.670999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109840,"kb_used_data":58696,"kb_used_omap":0,"kb_used_meta":51135,"kb_avail":94262000,"statfs":{"total":96636764160,"available":96524288000,"internally_reserved":0,"allocated":60104704,"data_stored":59946806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":52362292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.234}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.745000000000001}]}]}]}} 2021-01-15T18:41:15.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:15.029 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 75 20 38122 38102 3.96765 2.50781 0.0489824 0.0393287 2021-01-15T18:41:16.029 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 76 20 38478 38458 3.95202 2.78125 0.0425023 0.0395185 2021-01-15T18:41:16.460 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21507/151032 objects degraded (14.240%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":28,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37758,"data_bytes":309201172,"bytes_used":819363840,"bytes_avail":385727692800,"bytes_total":386547056640,"degraded_objects":21507,"degraded_total":151032,"degraded_ratio":0.14240028603209917,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":329551,"recovering_keys_per_sec":0,"num_objects_recovered":483,"num_bytes_recovered":3956736,"num_keys_recovered":0,"write_bytes_sec":3578665,"read_op_per_sec":0,"write_op_per_sec":436},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:16.497 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:16.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:17.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 77 20 38813 38793 3.93463 2.61719 0.0434336 0.0396885 2021-01-15T18:41:17.628 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:17.631 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:17.707 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2021-01-15T18:41:16.386421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309201172,"num_objects":37758,"num_object_clones":0,"num_object_copies":151032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37785,"num_write_kb":302070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37812,"ondisk_log_size":37812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":815128,"kb_used_data":516880,"kb_used_omap":2,"kb_used_meta":298301,"kb_avail":376672232,"statfs":{"total":386547056640,"available":385712365568,"internally_reserved":0,"allocated":529285120,"data_stored":528653528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":305460380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,14,20,27,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":64,"apply_latency_ms":64,"commit_latency_ns":64000000,"apply_latency_ns":64000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28393472,"num_objects":3466,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":27728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":2580480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006580"},"pg_stats":[{"pgid":"1.f","version":"28'2252","reported_seq":"2290","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.232133+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:14.232133+0000","last_peered":"2021-01-15T18:41:14.232133+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:14.232133+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2252,"ondisk_log_size":2252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18448384,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":18016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2342","reported_seq":"4726","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.786303+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:13.786303+0000","last_peered":"2021-01-15T18:41:13.786303+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:13.786303+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2342,"ondisk_log_size":2342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19144870,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":18708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2329","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.119061+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:13.119061+0000","last_peered":"2021-01-15T18:41:13.119061+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:13.119061+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2329,"ondisk_log_size":2329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19050579,"num_objects":2326,"num_object_clones":0,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2328,"num_write_kb":18610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":910}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2423","reported_seq":"2461","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.115004+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:14.115004+0000","last_peered":"2021-01-15T18:41:14.115004+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:14.115004+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2423,"ondisk_log_size":2423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19849216,"num_objects":2423,"num_object_clones":0,"num_object_copies":9692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2423,"num_write_kb":19384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2436","reported_seq":"2492","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.641558+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:13.641558+0000","last_peered":"2021-01-15T18:41:13.641558+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:13.641558+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2436,"ondisk_log_size":2436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19906726,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":19452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2303","reported_seq":"2373","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.192313+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:13.192313+0000","last_peered":"2021-01-15T18:41:13.192313+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:13.192313+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2303,"ondisk_log_size":2303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18833574,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2301,"num_write_kb":18399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2336","reported_seq":"2381","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.185552+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:14.185552+0000","last_peered":"2021-01-15T18:41:14.185552+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:14.185552+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2336,"ondisk_log_size":2336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19095718,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2334,"num_write_kb":18660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2386","reported_seq":"2452","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.123327+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:13.123327+0000","last_peered":"2021-01-15T18:41:13.123327+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:13.123327+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2386,"ondisk_log_size":2386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19533907,"num_objects":2385,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2406","reported_seq":"2438","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.258191+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:14.258191+0000","last_peered":"2021-01-15T18:41:14.258191+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:14.258191+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19697747,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2405,"num_write_kb":19237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2434","reported_seq":"2486","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.264460+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:14.264460+0000","last_peered":"2021-01-15T18:41:14.264460+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:14.264460+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2434,"ondisk_log_size":2434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19886329,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2280","reported_seq":"2330","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.221305+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:14.221305+0000","last_peered":"2021-01-15T18:41:14.221305+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:14.221305+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2280,"ondisk_log_size":2280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18645158,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2278,"num_write_kb":18215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2288","reported_seq":"2344","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.580993+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:13.580993+0000","last_peered":"2021-01-15T18:41:13.580993+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:13.580993+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2288,"ondisk_log_size":2288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18702502,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2286,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":913}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2314","reported_seq":"2371","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.773175+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:13.773175+0000","last_peered":"2021-01-15T18:41:13.773175+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:13.773175+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2314,"ondisk_log_size":2314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18899110,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":18468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2451","reported_seq":"2508","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.793840+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:13.793840+0000","last_peered":"2021-01-15T18:41:13.793840+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:13.793840+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2451,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":19572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2505","reported_seq":"2556","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:14.256874+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:14.256874+0000","last_peered":"2021-01-15T18:41:14.256874+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:14.256874+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2505,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20439372,"num_objects":2497,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":19979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2327","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:13.080733+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:13.080733+0000","last_peered":"2021-01-15T18:41:13.080733+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:13.080733+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2327,"ondisk_log_size":2327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19038374,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2325,"num_write_kb":18594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309201172,"num_objects":37758,"num_object_clones":0,"num_object_copies":151032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37785,"num_write_kb":302070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":531087360,"data_stored":531087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37812,"ondisk_log_size":37812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232208,"kb_used_data":150812,"kb_used_omap":0,"kb_used_meta":81471,"kb_avail":94139632,"statfs":{"total":96636764160,"available":96398983168,"internally_reserved":0,"allocated":154431488,"data_stored":154273590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":83426356},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":12.91},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":1.4730000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":223016,"kb_used_data":149988,"kb_used_omap":0,"kb_used_meta":73024,"kb_avail":94148824,"statfs":{"total":96636764160,"available":96408395776,"internally_reserved":0,"allocated":153587712,"data_stored":153429814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74776576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5840000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.7249999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2469999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":2.2570000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235096,"kb_used_data":149904,"kb_used_omap":0,"kb_used_meta":85183,"kb_avail":94136744,"statfs":{"total":96636764160,"available":96396025856,"internally_reserved":0,"allocated":153501696,"data_stored":153343798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87227444},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4159999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":7.202},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":14.670999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":124808,"kb_used_data":66176,"kb_used_omap":0,"kb_used_meta":58623,"kb_avail":94247032,"statfs":{"total":96636764160,"available":96508960768,"internally_reserved":0,"allocated":67764224,"data_stored":67606326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60030004},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3920000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.234}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]}]}} 2021-01-15T18:41:18.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 78 20 39170 39150 3.91994 2.78906 0.0498562 0.0398463 2021-01-15T18:41:19.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 79 20 39456 39436 3.89861 2.23438 0.0280171 0.0400588 2021-01-15T18:41:20.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:41:20.031461+0000 min lat: 0.00287162 max lat: 2.35257 avg lat: 0.0403081 2021-01-15T18:41:20.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:41:20.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 80 20 39712 39692 3.87488 2 0.0688641 0.0403081 2021-01-15T18:41:20.709 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:21.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 81 20 39970 39950 3.85193 2.01562 0.0473152 0.0405303 2021-01-15T18:41:21.641 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:21.658 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:21.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2021-01-15T18:41:20.783448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322652436,"num_objects":39400,"num_object_clones":0,"num_object_copies":157600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39427,"num_write_kb":315206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39454,"ondisk_log_size":39454,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":851280,"kb_used_data":544496,"kb_used_omap":2,"kb_used_meta":306749,"kb_avail":376636080,"statfs":{"total":386547056640,"available":385675345920,"internally_reserved":0,"allocated":557563904,"data_stored":556932312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":314111132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,16,13,18,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27852800,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":2777088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.400488"},"pg_stats":[{"pgid":"1.f","version":"28'2362","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.097390+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:19.097390+0000","last_peered":"2021-01-15T18:41:19.097390+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:19.097390+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2362,"ondisk_log_size":2362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19349504,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2438","reported_seq":"5192","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.828824+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:18.828824+0000","last_peered":"2021-01-15T18:41:18.828824+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:18.828824+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19931302,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2455","reported_seq":"2526","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.082272+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:18.082272+0000","last_peered":"2021-01-15T18:41:18.082272+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:18.082272+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2455,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20082771,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2511","reported_seq":"2549","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.156779+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:19.156779+0000","last_peered":"2021-01-15T18:41:19.156779+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:19.156779+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2511,"ondisk_log_size":2511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20570112,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2532","reported_seq":"2588","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.704821+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:18.704821+0000","last_peered":"2021-01-15T18:41:18.704821+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:18.704821+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2532,"ondisk_log_size":2532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20693158,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":20220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2417","reported_seq":"2488","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:17.894913+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:17.894913+0000","last_peered":"2021-01-15T18:41:17.894913+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:17.894913+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2417,"ondisk_log_size":2417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19767462,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":19311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2433","reported_seq":"2479","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.112326+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:19.112326+0000","last_peered":"2021-01-15T18:41:19.112326+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:19.112326+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19890342,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2492","reported_seq":"2557","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.073978+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:18.073978+0000","last_peered":"2021-01-15T18:41:18.073978+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:18.073978+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2492,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20402259,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":19925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2507","reported_seq":"2541","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.113909+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:19.113909+0000","last_peered":"2021-01-15T18:41:19.113909+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:19.113909+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2507,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":20045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2514","reported_seq":"2566","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.036355+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:19.036355+0000","last_peered":"2021-01-15T18:41:19.036355+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:19.036355+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2514,"ondisk_log_size":2514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20541689,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2390","reported_seq":"2439","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.165847+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:19.165847+0000","last_peered":"2021-01-15T18:41:19.165847+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:19.165847+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2390,"ondisk_log_size":2390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19546278,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":19095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2387","reported_seq":"2443","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.760494+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:18.760494+0000","last_peered":"2021-01-15T18:41:18.760494+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:18.760494+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2387,"ondisk_log_size":2387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19513510,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2408","reported_seq":"2466","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.705068+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:18.705068+0000","last_peered":"2021-01-15T18:41:18.705068+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:18.705068+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19669158,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":997}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2558","reported_seq":"2615","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.687748+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:18.687748+0000","last_peered":"2021-01-15T18:41:18.687748+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:18.687748+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2558,"ondisk_log_size":2558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20906150,"num_objects":2553,"num_object_clones":0,"num_object_copies":10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2612","reported_seq":"2664","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.158864+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:19.158864+0000","last_peered":"2021-01-15T18:41:19.158864+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:19.158864+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2612,"ondisk_log_size":2612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21315916,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":20835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2438","reported_seq":"2511","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.086182+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:18.086182+0000","last_peered":"2021-01-15T18:41:18.086182+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:18.086182+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19947686,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":322652436,"num_objects":39400,"num_object_clones":0,"num_object_copies":157600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39427,"num_write_kb":315206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558972928,"data_stored":558972928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39454,"ondisk_log_size":39454,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245912,"kb_used_data":157392,"kb_used_omap":0,"kb_used_meta":88511,"kb_avail":94125928,"statfs":{"total":96636764160,"available":96384950272,"internally_reserved":0,"allocated":161169408,"data_stored":161011510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":90635316},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":12.91},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":1.4730000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241436,"kb_used_data":157012,"kb_used_omap":0,"kb_used_meta":84416,"kb_avail":94130404,"statfs":{"total":96636764160,"available":96389533696,"internally_reserved":0,"allocated":160780288,"data_stored":160622390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86441984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5840000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.7249999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2469999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":2.2570000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280535,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236776,"kb_used_data":156704,"kb_used_omap":0,"kb_used_meta":80063,"kb_avail":94135064,"statfs":{"total":96636764160,"available":96394305536,"internally_reserved":0,"allocated":160464896,"data_stored":160306998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81984564},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.641999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":1.137},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":2.1640000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":12.064}]}]},{"osd":1,"up_from":26,"seq":111669149707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127156,"kb_used_data":73388,"kb_used_omap":0,"kb_used_meta":53759,"kb_avail":94244684,"statfs":{"total":96636764160,"available":96506556416,"internally_reserved":0,"allocated":75149312,"data_stored":74991414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55049268},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.541}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:21.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:22.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 82 20 40178 40158 3.82478 1.625 0.013015 0.0405112 2021-01-15T18:41:22.123 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:22.124 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21321/157600 objects degraded (13.529%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":132,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39400,"data_bytes":322652436,"bytes_used":871710720,"bytes_avail":385675345920,"bytes_total":386547056640,"degraded_objects":21321,"degraded_total":157600,"degraded_ratio":0.13528553299492385,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":341858,"recovering_keys_per_sec":0,"num_objects_recovered":501,"num_bytes_recovered":4104192,"num_keys_recovered":0,"write_bytes_sec":3211149,"read_op_per_sec":0,"write_op_per_sec":391},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:22.136 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:22.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:22.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:22.415 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:22.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2021-01-15T18:41:22.388414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322652436,"num_objects":39400,"num_object_clones":0,"num_object_copies":157600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39427,"num_write_kb":315206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39454,"ondisk_log_size":39454,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":851280,"kb_used_data":544496,"kb_used_omap":2,"kb_used_meta":306749,"kb_avail":376636080,"statfs":{"total":386547056640,"available":385675345920,"internally_reserved":0,"allocated":557563904,"data_stored":556932312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":314111132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,16,13,18,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13451264,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":13136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002516"},"pg_stats":[{"pgid":"1.f","version":"28'2362","reported_seq":"2400","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.097390+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:19.097390+0000","last_peered":"2021-01-15T18:41:19.097390+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:19.097390+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2362,"ondisk_log_size":2362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19349504,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1008}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2438","reported_seq":"5192","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.828824+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:18.828824+0000","last_peered":"2021-01-15T18:41:18.828824+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:18.828824+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19931302,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2455","reported_seq":"2526","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.082272+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:18.082272+0000","last_peered":"2021-01-15T18:41:18.082272+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:18.082272+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2455,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20082771,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"28'2511","reported_seq":"2549","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.156779+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:19.156779+0000","last_peered":"2021-01-15T18:41:19.156779+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:19.156779+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2511,"ondisk_log_size":2511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20570112,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"28'2532","reported_seq":"2588","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.704821+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:18.704821+0000","last_peered":"2021-01-15T18:41:18.704821+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:18.704821+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2532,"ondisk_log_size":2532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20693158,"num_objects":2527,"num_object_clones":0,"num_object_copies":10108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":20220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"28'2417","reported_seq":"2488","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:17.894913+0000","last_change":"2021-01-15T18:40:32.710974+0000","last_active":"2021-01-15T18:41:17.894913+0000","last_peered":"2021-01-15T18:41:17.894913+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:17.894913+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2417,"ondisk_log_size":2417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19767462,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":19311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"28'2433","reported_seq":"2479","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.112326+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:19.112326+0000","last_peered":"2021-01-15T18:41:19.112326+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:19.112326+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19890342,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2492","reported_seq":"2557","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.073978+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:18.073978+0000","last_peered":"2021-01-15T18:41:18.073978+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:18.073978+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2492,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20402259,"num_objects":2491,"num_object_clones":0,"num_object_copies":9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":19925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"28'2507","reported_seq":"2541","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.113909+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:19.113909+0000","last_peered":"2021-01-15T18:41:19.113909+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:19.113909+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2507,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":20045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1070}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"28'2514","reported_seq":"2566","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.036355+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:19.036355+0000","last_peered":"2021-01-15T18:41:19.036355+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:19.036355+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2514,"ondisk_log_size":2514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20541689,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"28'2390","reported_seq":"2439","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.165847+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:19.165847+0000","last_peered":"2021-01-15T18:41:19.165847+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:19.165847+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2390,"ondisk_log_size":2390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19546278,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2388,"num_write_kb":19095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'2387","reported_seq":"2443","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.760494+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:18.760494+0000","last_peered":"2021-01-15T18:41:18.760494+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:18.760494+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2387,"ondisk_log_size":2387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19513510,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":19063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"28'2408","reported_seq":"2466","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.705068+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:18.705068+0000","last_peered":"2021-01-15T18:41:18.705068+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:18.705068+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19669158,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":997}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2558","reported_seq":"2615","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.687748+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:18.687748+0000","last_peered":"2021-01-15T18:41:18.687748+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:18.687748+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2558,"ondisk_log_size":2558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20906150,"num_objects":2553,"num_object_clones":0,"num_object_copies":10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2612","reported_seq":"2664","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:19.158864+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:19.158864+0000","last_peered":"2021-01-15T18:41:19.158864+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:19.158864+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2612,"ondisk_log_size":2612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21315916,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":20835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2438","reported_seq":"2511","reported_epoch":"28","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.086182+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:18.086182+0000","last_peered":"2021-01-15T18:41:18.086182+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:18.086182+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19947686,"num_objects":2436,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1054}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":322652436,"num_objects":39400,"num_object_clones":0,"num_object_copies":157600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39427,"num_write_kb":315206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558972928,"data_stored":558972928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39454,"ondisk_log_size":39454,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215126,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245912,"kb_used_data":157392,"kb_used_omap":0,"kb_used_meta":88511,"kb_avail":94125928,"statfs":{"total":96636764160,"available":96384950272,"internally_reserved":0,"allocated":161169408,"data_stored":161011510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":90635316},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":12.91},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":1.4730000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241436,"kb_used_data":157012,"kb_used_omap":0,"kb_used_meta":84416,"kb_avail":94130404,"statfs":{"total":96636764160,"available":96389533696,"internally_reserved":0,"allocated":160780288,"data_stored":160622390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86441984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5840000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":4.7249999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.2469999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":2.2570000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280535,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236776,"kb_used_data":156704,"kb_used_omap":0,"kb_used_meta":80063,"kb_avail":94135064,"statfs":{"total":96636764160,"available":96394305536,"internally_reserved":0,"allocated":160464896,"data_stored":160306998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81984564},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.641999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":1.137},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":2.1640000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":12.064}]}]},{"osd":1,"up_from":26,"seq":111669149707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127156,"kb_used_data":73388,"kb_used_omap":0,"kb_used_meta":53759,"kb_avail":94244684,"statfs":{"total":96636764160,"available":96506556416,"internally_reserved":0,"allocated":75149312,"data_stored":74991414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55049268},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.541}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:23.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 83 20 40178 40158 3.77871 0 - 0.0405112 2021-01-15T18:41:24.031 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 84 20 40178 40158 3.73373 0 - 0.0405112 2021-01-15T18:41:25.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 85 20 40178 40158 3.68981 0 - 0.0405112 2021-01-15T18:41:25.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:25.699 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:25.701 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:25.711 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2021-01-15T18:41:24.388945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328911124,"num_objects":40164,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40191,"num_write_kb":321318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":11567104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":872584,"kb_used_data":556072,"kb_used_omap":2,"kb_used_meta":316477,"kb_avail":376614776,"statfs":{"total":386547056640,"available":385653530624,"internally_reserved":0,"allocated":569417728,"data_stored":568758551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":324072604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,5,11,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2056192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002524"},"pg_stats":[{"pgid":"1.f","version":"29'2406","reported_seq":"2446","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.700515+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:21.700515+0000","last_peered":"2021-01-15T18:41:21.700515+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:21.700515+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2438","reported_seq":"5192","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.828824+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:18.828824+0000","last_peered":"2021-01-15T18:41:18.828824+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:18.828824+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19931302,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2509","reported_seq":"2582","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.558311+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:21.558311+0000","last_peered":"2021-01-15T18:41:21.558311+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:21.558311+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2509,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2560","reported_seq":"2600","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.800327+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.800327+0000","last_peered":"2021-01-15T18:41:21.800327+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.800327+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2560,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2689","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.936874+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:21.936874+0000","last_peered":"2021-01-15T18:41:21.936874+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:21.936874+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2484","reported_seq":"2532","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.799086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:21.799086+0000","last_peered":"2021-01-15T18:41:21.799086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:21.799086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2484,"ondisk_log_size":2484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20308134,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2560","reported_seq":"2596","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.896594+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:21.896594+0000","last_peered":"2021-01-15T18:41:21.896594+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:21.896594+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20959315,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2559","reported_seq":"2613","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.793059+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.793059+0000","last_peered":"2021-01-15T18:41:21.793059+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.793059+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20910329,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2431","reported_seq":"2483","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.888143+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:21.888143+0000","last_peered":"2021-01-15T18:41:21.888143+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:21.888143+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2431","reported_seq":"2489","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.783185+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:21.783185+0000","last_peered":"2021-01-15T18:41:21.783185+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:21.783185+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19873958,"num_objects":2427,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"29'2454","reported_seq":"2514","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.792158+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:21.792158+0000","last_peered":"2021-01-15T18:41:21.792158+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:21.792158+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2454,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20045990,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1043}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2623","reported_seq":"2682","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.869989+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.869989+0000","last_peered":"2021-01-15T18:41:21.869989+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.869989+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2623,"ondisk_log_size":2623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21438630,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2658","reported_seq":"2712","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.746457+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.746457+0000","last_peered":"2021-01-15T18:41:21.746457+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.746457+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2658,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21692748,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2495","reported_seq":"2570","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.809894+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:21.809894+0000","last_peered":"2021-01-15T18:41:21.809894+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:21.809894+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2495,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328911124,"num_objects":40164,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40191,"num_write_kb":321318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":11567104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":568815616,"data_stored":568815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242948,"kb_used_data":160892,"kb_used_omap":0,"kb_used_meta":82047,"kb_avail":94128892,"statfs":{"total":96636764160,"available":96387985408,"internally_reserved":0,"allocated":164753408,"data_stored":164586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84016180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.872},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":3.8999999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.52500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255368,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":94464,"kb_avail":94116472,"statfs":{"total":96636764160,"available":96375267328,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96731136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":16.597999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":7.0819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2590000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247112,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":86207,"kb_avail":94124728,"statfs":{"total":96636764160,"available":96383721472,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88276020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.92700000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.876}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.1040000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127156,"kb_used_data":73388,"kb_used_omap":0,"kb_used_meta":53759,"kb_avail":94244684,"statfs":{"total":96636764160,"available":96506556416,"internally_reserved":0,"allocated":75149312,"data_stored":74991414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55049268},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.541}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:25.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:26.018 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:26.019 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21254/160656 objects degraded (13.230%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":136,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":40164,"data_bytes":328911124,"bytes_used":893526016,"bytes_avail":385653530624,"bytes_total":386547056640,"degraded_objects":21254,"degraded_total":160656,"degraded_ratio":0.13229509013046509,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":256942,"recovering_keys_per_sec":0,"num_objects_recovered":251,"num_bytes_recovered":2056192,"num_keys_recovered":0,"write_bytes_sec":2462966,"read_op_per_sec":0,"write_op_per_sec":300},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:26.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 86 20 40178 40158 3.64692 0 - 0.0405112 2021-01-15T18:41:26.032 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:26.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:26.289 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:26.290 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:26.301 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2021-01-15T18:41:24.388945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328911124,"num_objects":40164,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40191,"num_write_kb":321318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":11567104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":872584,"kb_used_data":556072,"kb_used_omap":2,"kb_used_meta":316477,"kb_avail":376614776,"statfs":{"total":386547056640,"available":385653530624,"internally_reserved":0,"allocated":569417728,"data_stored":568758551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":324072604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,5,11,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2056192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002524"},"pg_stats":[{"pgid":"1.f","version":"29'2406","reported_seq":"2446","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.700515+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:21.700515+0000","last_peered":"2021-01-15T18:41:21.700515+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:21.700515+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2438","reported_seq":"5192","reported_epoch":"28","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:18.828824+0000","last_change":"2021-01-15T18:40:32.664795+0000","last_active":"2021-01-15T18:41:18.828824+0000","last_peered":"2021-01-15T18:41:18.828824+0000","last_clean":"2021-01-15T18:39:34.208471+0000","last_became_active":"2021-01-15T18:40:32.637456+0000","last_became_peered":"2021-01-15T18:40:32.637456+0000","last_unstale":"2021-01-15T18:41:18.828824+0000","last_undegraded":"2021-01-15T18:40:32.519616+0000","last_fullsized":"2021-01-15T18:40:32.487725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19931302,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":19476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":11034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"29'2509","reported_seq":"2582","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.558311+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:21.558311+0000","last_peered":"2021-01-15T18:41:21.558311+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:21.558311+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2509,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2560","reported_seq":"2600","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.800327+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.800327+0000","last_peered":"2021-01-15T18:41:21.800327+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.800327+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2560,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2689","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.936874+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:21.936874+0000","last_peered":"2021-01-15T18:41:21.936874+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:21.936874+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2484","reported_seq":"2532","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.799086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:21.799086+0000","last_peered":"2021-01-15T18:41:21.799086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:21.799086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2484,"ondisk_log_size":2484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20308134,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2560","reported_seq":"2596","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.896594+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:21.896594+0000","last_peered":"2021-01-15T18:41:21.896594+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:21.896594+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20959315,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2559","reported_seq":"2613","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.793059+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.793059+0000","last_peered":"2021-01-15T18:41:21.793059+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.793059+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20910329,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2431","reported_seq":"2483","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.888143+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:21.888143+0000","last_peered":"2021-01-15T18:41:21.888143+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:21.888143+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2431","reported_seq":"2489","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.783185+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:21.783185+0000","last_peered":"2021-01-15T18:41:21.783185+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:21.783185+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19873958,"num_objects":2427,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"29'2454","reported_seq":"2514","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.792158+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:21.792158+0000","last_peered":"2021-01-15T18:41:21.792158+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:21.792158+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2454,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20045990,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1043}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2623","reported_seq":"2682","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.869989+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.869989+0000","last_peered":"2021-01-15T18:41:21.869989+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.869989+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2623,"ondisk_log_size":2623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21438630,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2658","reported_seq":"2712","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.746457+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.746457+0000","last_peered":"2021-01-15T18:41:21.746457+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.746457+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2658,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21692748,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2495","reported_seq":"2570","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.809894+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:21.809894+0000","last_peered":"2021-01-15T18:41:21.809894+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:21.809894+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2495,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328911124,"num_objects":40164,"num_object_clones":0,"num_object_copies":160656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40191,"num_write_kb":321318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":11567104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":568815616,"data_stored":568815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40218,"ondisk_log_size":40218,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242948,"kb_used_data":160892,"kb_used_omap":0,"kb_used_meta":82047,"kb_avail":94128892,"statfs":{"total":96636764160,"available":96387985408,"internally_reserved":0,"allocated":164753408,"data_stored":164586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84016180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.872},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":3.8999999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.52500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255368,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":94464,"kb_avail":94116472,"statfs":{"total":96636764160,"available":96375267328,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96731136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":16.597999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":7.0819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2590000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247112,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":86207,"kb_avail":94124728,"statfs":{"total":96636764160,"available":96383721472,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88276020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.92700000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.876}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.1040000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127156,"kb_used_data":73388,"kb_used_omap":0,"kb_used_meta":53759,"kb_avail":94244684,"statfs":{"total":96636764160,"available":96506556416,"internally_reserved":0,"allocated":75149312,"data_stored":74991414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":55049268},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.541}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:27.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 87 20 40178 40158 3.605 0 - 0.0405112 2021-01-15T18:41:28.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 88 20 40178 40158 3.56405 0 - 0.0405112 2021-01-15T18:41:29.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 89 20 40192 40172 3.52524 0.015625 0.198666 0.0408456 2021-01-15T18:41:29.304 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:29.616 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:29.617 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:29.630 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-01-15T18:41:28.389859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329099540,"num_objects":40187,"num_object_clones":0,"num_object_copies":160748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40214,"num_write_kb":321502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":881120,"kb_used_data":559040,"kb_used_omap":2,"kb_used_meta":322045,"kb_avail":376606240,"statfs":{"total":386547056640,"available":385644789760,"internally_reserved":0,"allocated":572456960,"data_stored":571788588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":329774236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2,0,0,2,9,7,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6447104,"num_objects":787,"num_object_clones":0,"num_object_copies":3148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":787,"num_write_kb":6296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001948"},"pg_stats":[{"pgid":"1.f","version":"29'2406","reported_seq":"2446","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.700515+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:21.700515+0000","last_peered":"2021-01-15T18:41:21.700515+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:21.700515+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"29'2509","reported_seq":"2582","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.558311+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:21.558311+0000","last_peered":"2021-01-15T18:41:21.558311+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:21.558311+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2509,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2560","reported_seq":"2600","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.800327+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.800327+0000","last_peered":"2021-01-15T18:41:21.800327+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.800327+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2560,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2689","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.936874+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:21.936874+0000","last_peered":"2021-01-15T18:41:21.936874+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:21.936874+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2484","reported_seq":"2532","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.799086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:21.799086+0000","last_peered":"2021-01-15T18:41:21.799086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:21.799086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2484,"ondisk_log_size":2484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20308134,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2560","reported_seq":"2596","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.896594+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:21.896594+0000","last_peered":"2021-01-15T18:41:21.896594+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:21.896594+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20959315,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2559","reported_seq":"2613","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.793059+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.793059+0000","last_peered":"2021-01-15T18:41:21.793059+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.793059+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20910329,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2431","reported_seq":"2483","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.888143+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:21.888143+0000","last_peered":"2021-01-15T18:41:21.888143+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:21.888143+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2431","reported_seq":"2489","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.783185+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:21.783185+0000","last_peered":"2021-01-15T18:41:21.783185+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:21.783185+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19873958,"num_objects":2427,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"29'2454","reported_seq":"2514","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.792158+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:21.792158+0000","last_peered":"2021-01-15T18:41:21.792158+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:21.792158+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2454,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20045990,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1043}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2623","reported_seq":"2682","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.869989+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.869989+0000","last_peered":"2021-01-15T18:41:21.869989+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.869989+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2623,"ondisk_log_size":2623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21438630,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2658","reported_seq":"2712","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.746457+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.746457+0000","last_peered":"2021-01-15T18:41:21.746457+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.746457+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2658,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21692748,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2495","reported_seq":"2570","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.809894+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:21.809894+0000","last_peered":"2021-01-15T18:41:21.809894+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:21.809894+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2495,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329099540,"num_objects":40187,"num_object_clones":0,"num_object_copies":160748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40214,"num_write_kb":321502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571441152,"data_stored":571441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242948,"kb_used_data":160892,"kb_used_omap":0,"kb_used_meta":82047,"kb_avail":94128892,"statfs":{"total":96636764160,"available":96387985408,"internally_reserved":0,"allocated":164753408,"data_stored":164586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84016180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.872},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":3.8999999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.52500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255368,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":94464,"kb_avail":94116472,"statfs":{"total":96636764160,"available":96375267328,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96731136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2590000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247112,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":86207,"kb_avail":94124728,"statfs":{"total":96636764160,"available":96383721472,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88276020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.92700000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.876}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.1040000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135692,"kb_used_data":76356,"kb_used_omap":0,"kb_used_meta":59327,"kb_avail":94236148,"statfs":{"total":96636764160,"available":96497815552,"internally_reserved":0,"allocated":78188544,"data_stored":78021451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60750900},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,9,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}]}} 2021-01-15T18:41:29.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:30.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:30.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21183/160748 objects degraded (13.178%), 15 pgs degraded","count":15},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":140,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean+wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40187,"data_bytes":329099540,"bytes_used":902266880,"bytes_avail":385644789760,"bytes_total":386547056640,"degraded_objects":21183,"degraded_total":160748,"degraded_ratio":0.13177768930251077,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":139230,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"write_bytes_sec":805691,"read_op_per_sec":0,"write_op_per_sec":98},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-01-15T18:40:50.375858+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:41:30.018 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:30.019 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:30.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 90 20 40201 40181 3.48686 0.0703125 0.00851576 0.0408384 2021-01-15T18:41:30.339 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:30.342 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:30.353 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-01-15T18:41:28.389859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329099540,"num_objects":40187,"num_object_clones":0,"num_object_copies":160748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40214,"num_write_kb":321502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":881120,"kb_used_data":559040,"kb_used_omap":2,"kb_used_meta":322045,"kb_avail":376606240,"statfs":{"total":386547056640,"available":385644789760,"internally_reserved":0,"allocated":572456960,"data_stored":571788588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":329774236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2,0,0,2,9,7,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6447104,"num_objects":787,"num_object_clones":0,"num_object_copies":3148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":787,"num_write_kb":6296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001948"},"pg_stats":[{"pgid":"1.f","version":"29'2406","reported_seq":"2446","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.700515+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:21.700515+0000","last_peered":"2021-01-15T18:41:21.700515+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:21.700515+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2406,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19709952,"num_objects":2406,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":19248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"29'2509","reported_seq":"2582","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.558311+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:21.558311+0000","last_peered":"2021-01-15T18:41:21.558311+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:21.558311+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2509,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2560","reported_seq":"2600","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.800327+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.800327+0000","last_peered":"2021-01-15T18:41:21.800327+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.800327+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":2560,"num_object_clones":0,"num_object_copies":10240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2560,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2689","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.936874+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:21.936874+0000","last_peered":"2021-01-15T18:41:21.936874+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:21.936874+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2484","reported_seq":"2532","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.799086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:21.799086+0000","last_peered":"2021-01-15T18:41:21.799086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:21.799086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2484,"ondisk_log_size":2484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20308134,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":19844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2560","reported_seq":"2596","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.896594+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:21.896594+0000","last_peered":"2021-01-15T18:41:21.896594+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:21.896594+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20959315,"num_objects":2559,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":20469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1123}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2559","reported_seq":"2613","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.793059+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.793059+0000","last_peered":"2021-01-15T18:41:21.793059+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.793059+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20910329,"num_objects":2554,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2431","reported_seq":"2483","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.888143+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:21.888143+0000","last_peered":"2021-01-15T18:41:21.888143+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:21.888143+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2431","reported_seq":"2489","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.783185+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:21.783185+0000","last_peered":"2021-01-15T18:41:21.783185+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:21.783185+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19873958,"num_objects":2427,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"29'2454","reported_seq":"2514","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.792158+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:21.792158+0000","last_peered":"2021-01-15T18:41:21.792158+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:21.792158+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2454,"ondisk_log_size":2454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20045990,"num_objects":2448,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1043}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2623","reported_seq":"2682","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.869989+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.869989+0000","last_peered":"2021-01-15T18:41:21.869989+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.869989+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2623,"ondisk_log_size":2623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21438630,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":20948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"29'2658","reported_seq":"2712","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.746457+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.746457+0000","last_peered":"2021-01-15T18:41:21.746457+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.746457+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2658,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21692748,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2495","reported_seq":"2570","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.809894+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:21.809894+0000","last_peered":"2021-01-15T18:41:21.809894+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:21.809894+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2495,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329099540,"num_objects":40187,"num_object_clones":0,"num_object_copies":160748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40214,"num_write_kb":321502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571441152,"data_stored":571441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242948,"kb_used_data":160892,"kb_used_omap":0,"kb_used_meta":82047,"kb_avail":94128892,"statfs":{"total":96636764160,"available":96387985408,"internally_reserved":0,"allocated":164753408,"data_stored":164586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84016180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.872},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":3.8999999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.52500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255368,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":94464,"kb_avail":94116472,"statfs":{"total":96636764160,"available":96375267328,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96731136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2590000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247112,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":86207,"kb_avail":94124728,"statfs":{"total":96636764160,"available":96383721472,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88276020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.92700000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.876}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.999},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.1040000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135692,"kb_used_data":76356,"kb_used_omap":0,"kb_used_meta":59327,"kb_avail":94236148,"statfs":{"total":96636764160,"available":96497815552,"internally_reserved":0,"allocated":78188544,"data_stored":78021451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60750900},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,9,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}]}} 2021-01-15T18:41:31.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 91 20 40201 40181 3.44855 0 - 0.0408384 2021-01-15T18:41:32.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 92 20 40201 40181 3.41107 0 - 0.0408384 2021-01-15T18:41:33.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 93 20 40201 40181 3.3744 0 - 0.0408384 2021-01-15T18:41:33.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:33.689 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:33.703 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2021-01-15T18:41:32.390619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329214228,"num_objects":40201,"num_object_clones":0,"num_object_copies":160804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40228,"num_write_kb":321614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40255,"ondisk_log_size":40255,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":881596,"kb_used_data":559388,"kb_used_omap":2,"kb_used_meta":322173,"kb_avail":376605764,"statfs":{"total":386547056640,"available":385644302336,"internally_reserved":0,"allocated":572813312,"data_stored":572144940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":329905308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,0,0,0,0,0,0,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6561792,"num_objects":801,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":6408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.612866"},"pg_stats":[{"pgid":"1.f","version":"30'2408","reported_seq":"2448","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.745771+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:28.745771+0000","last_peered":"2021-01-15T18:41:28.745771+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:28.745771+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19726336,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":19264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"29'2509","reported_seq":"2582","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.558311+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:21.558311+0000","last_peered":"2021-01-15T18:41:21.558311+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:21.558311+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2509,"ondisk_log_size":2509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20525139,"num_objects":2506,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2561","reported_seq":"2601","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.849265+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.849265+0000","last_peered":"2021-01-15T18:41:21.849265+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.849265+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2561,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20979712,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2689","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.936874+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:21.936874+0000","last_peered":"2021-01-15T18:41:21.936874+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:21.936874+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2486","reported_seq":"2534","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.773579+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:28.773579+0000","last_peered":"2021-01-15T18:41:28.773579+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:28.773579+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2486,"ondisk_log_size":2486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1090}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2562","reported_seq":"2598","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.819311+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:28.819311+0000","last_peered":"2021-01-15T18:41:28.819311+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:28.819311+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2562,"ondisk_log_size":2562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20975699,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2560","reported_seq":"2614","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.906913+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.906913+0000","last_peered":"2021-01-15T18:41:21.906913+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.906913+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20918521,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2433","reported_seq":"2485","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.796392+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:28.796392+0000","last_peered":"2021-01-15T18:41:28.796392+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:28.796392+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19898534,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2432","reported_seq":"2490","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.869205+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:21.869205+0000","last_peered":"2021-01-15T18:41:21.869205+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:21.869205+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2432,"ondisk_log_size":2432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":19423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"30'2455","reported_seq":"2515","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.794023+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:21.794023+0000","last_peered":"2021-01-15T18:41:21.794023+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:21.794023+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2455,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20054182,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":19596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1044}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2624","reported_seq":"2683","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.923308+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.923308+0000","last_peered":"2021-01-15T18:41:21.923308+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.923308+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2624,"ondisk_log_size":2624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21446822,"num_objects":2619,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2659","reported_seq":"2713","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.796505+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.796505+0000","last_peered":"2021-01-15T18:41:21.796505+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.796505+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2659,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21700940,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2495","reported_seq":"2570","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.809894+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:21.809894+0000","last_peered":"2021-01-15T18:41:21.809894+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:21.809894+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2495,"ondisk_log_size":2495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329214228,"num_objects":40201,"num_object_clones":0,"num_object_copies":160804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40228,"num_write_kb":321614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":12148736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":572141568,"data_stored":572141568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40255,"ondisk_log_size":40255,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242948,"kb_used_data":160892,"kb_used_omap":0,"kb_used_meta":82047,"kb_avail":94128892,"statfs":{"total":96636764160,"available":96387985408,"internally_reserved":0,"allocated":164753408,"data_stored":164586315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84016180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.38900000000000001},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6429999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.69499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255368,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":94464,"kb_avail":94116472,"statfs":{"total":96636764160,"available":96375267328,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96731136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2590000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247112,"kb_used_data":160896,"kb_used_omap":0,"kb_used_meta":86207,"kb_avail":94124728,"statfs":{"total":96636764160,"available":96383721472,"internally_reserved":0,"allocated":164757504,"data_stored":164590411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88276020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.58899999999999997},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.53100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.871},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.81200000000000006}]}]},{"osd":1,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136168,"kb_used_data":76704,"kb_used_omap":0,"kb_used_meta":59455,"kb_avail":94235672,"statfs":{"total":96636764160,"available":96497328128,"internally_reserved":0,"allocated":78544896,"data_stored":78377803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60881972},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,18],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]}]}} 2021-01-15T18:41:33.705 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:34.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 94 20 40201 40181 3.33851 0 - 0.0408384 2021-01-15T18:41:34.088 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:34.089 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21183/160804 objects degraded (13.173%), 15 pgs degraded","count":15},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":144,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40201,"data_bytes":329214228,"bytes_used":902754304,"bytes_avail":385644302336,"bytes_total":386547056640,"degraded_objects":21183,"degraded_total":160804,"degraded_ratio":0.13173179771647472,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":104977,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"write_bytes_sec":618286,"read_op_per_sec":0,"write_op_per_sec":75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:34.103 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:34.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:34.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:34.429 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2021-01-15T18:41:34.391010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329287956,"num_objects":40210,"num_object_clones":0,"num_object_copies":160840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40237,"num_write_kb":321686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":865660,"kb_used_data":559644,"kb_used_omap":2,"kb_used_meta":305981,"kb_avail":376621700,"statfs":{"total":386547056640,"available":385660620800,"internally_reserved":0,"allocated":573075456,"data_stored":572407084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":313324700},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,18],"upper_bound":16384},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6635520,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":6480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":7643136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002596"},"pg_stats":[{"pgid":"1.f","version":"30'2408","reported_seq":"2448","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.745771+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:28.745771+0000","last_peered":"2021-01-15T18:41:28.745771+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:28.745771+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19726336,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":19264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"30'2513","reported_seq":"2586","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.818018+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:28.818018+0000","last_peered":"2021-01-15T18:41:28.818018+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:28.818018+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20557907,"num_objects":2510,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2561","reported_seq":"2601","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.849265+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.849265+0000","last_peered":"2021-01-15T18:41:21.849265+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.849265+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2561,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20979712,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"4287","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:33.202640+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:33.202640+0000","last_peered":"2021-01-15T18:41:33.202640+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:33.202640+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2486","reported_seq":"2534","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.773579+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:28.773579+0000","last_peered":"2021-01-15T18:41:28.773579+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:28.773579+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2486,"ondisk_log_size":2486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1090}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2562","reported_seq":"2598","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.819311+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:28.819311+0000","last_peered":"2021-01-15T18:41:28.819311+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:28.819311+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2562,"ondisk_log_size":2562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20975699,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2560","reported_seq":"2614","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.906913+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.906913+0000","last_peered":"2021-01-15T18:41:21.906913+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.906913+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20918521,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2433","reported_seq":"2485","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.796392+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:28.796392+0000","last_peered":"2021-01-15T18:41:28.796392+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:28.796392+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19898534,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2433","reported_seq":"2491","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.782262+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:28.782262+0000","last_peered":"2021-01-15T18:41:28.782262+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:28.782262+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19890342,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"30'2456","reported_seq":"2516","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.760038+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:28.760038+0000","last_peered":"2021-01-15T18:41:28.760038+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:28.760038+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2456,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20062374,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1045}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2624","reported_seq":"2683","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.923308+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.923308+0000","last_peered":"2021-01-15T18:41:21.923308+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.923308+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2624,"ondisk_log_size":2624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21446822,"num_objects":2619,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2659","reported_seq":"2713","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.796505+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.796505+0000","last_peered":"2021-01-15T18:41:21.796505+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.796505+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2659,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21700940,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2498","reported_seq":"2573","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.828886+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:28.828886+0000","last_peered":"2021-01-15T18:41:28.828886+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:28.828886+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2498,"ondisk_log_size":2498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20439206,"num_objects":2496,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1114}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329287956,"num_objects":40210,"num_object_clones":0,"num_object_copies":160840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40237,"num_write_kb":321686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":572268544,"data_stored":572268544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243100,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":82111,"kb_avail":94128740,"statfs":{"total":96636764160,"available":96387829760,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84081716},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.627},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239132,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":78144,"kb_avail":94132708,"statfs":{"total":96636764160,"available":96391892992,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80019456},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280539,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247260,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":86271,"kb_avail":94124580,"statfs":{"total":96636764160,"available":96383569920,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88341556},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.79300000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136168,"kb_used_data":76704,"kb_used_omap":0,"kb_used_meta":59455,"kb_avail":94235672,"statfs":{"total":96636764160,"available":96497328128,"internally_reserved":0,"allocated":78544896,"data_stored":78377803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60881972},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,18],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]}]}} 2021-01-15T18:41:35.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 95 20 40201 40181 3.30338 0 - 0.0408384 2021-01-15T18:41:36.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 96 20 40201 40181 3.26898 0 - 0.0408384 2021-01-15T18:41:37.032 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 97 20 40308 40288 3.2439 0.11942 0.00769985 0.0480936 2021-01-15T18:41:37.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:37.883 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:37.896 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2021-01-15T18:41:36.516197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329287956,"num_objects":40210,"num_object_clones":0,"num_object_copies":160840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40237,"num_write_kb":321686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":879264,"kb_used_data":563328,"kb_used_omap":2,"kb_used_meta":315901,"kb_avail":376608096,"statfs":{"total":386547056640,"available":385646690304,"internally_reserved":0,"allocated":576847872,"data_stored":576179500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":323482780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,2,18],"upper_bound":32768},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":188416,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":6529024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.126338"},"pg_stats":[{"pgid":"1.f","version":"30'2408","reported_seq":"2448","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.745771+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:28.745771+0000","last_peered":"2021-01-15T18:41:28.745771+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:28.745771+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19726336,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":19264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"30'2513","reported_seq":"2586","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.818018+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:28.818018+0000","last_peered":"2021-01-15T18:41:28.818018+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:28.818018+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20557907,"num_objects":2510,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2561","reported_seq":"2601","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.849265+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.849265+0000","last_peered":"2021-01-15T18:41:21.849265+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.849265+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2561,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20979712,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"4287","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:33.202640+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:33.202640+0000","last_peered":"2021-01-15T18:41:33.202640+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:33.202640+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2486","reported_seq":"2534","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.773579+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:28.773579+0000","last_peered":"2021-01-15T18:41:28.773579+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:28.773579+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2486,"ondisk_log_size":2486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1090}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"29'2547","reported_seq":"2615","reported_epoch":"29","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.652315+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:21.652315+0000","last_peered":"2021-01-15T18:41:21.652315+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:21.652315+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2547,"ondisk_log_size":2547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852819,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":20365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1094}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2562","reported_seq":"2598","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.819311+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:28.819311+0000","last_peered":"2021-01-15T18:41:28.819311+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:28.819311+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2562,"ondisk_log_size":2562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20975699,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2560","reported_seq":"2614","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.906913+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.906913+0000","last_peered":"2021-01-15T18:41:21.906913+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.906913+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20918521,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2433","reported_seq":"2485","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.796392+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:28.796392+0000","last_peered":"2021-01-15T18:41:28.796392+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:28.796392+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19898534,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2433","reported_seq":"2491","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.782262+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:28.782262+0000","last_peered":"2021-01-15T18:41:28.782262+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:28.782262+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19890342,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"30'2456","reported_seq":"2516","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.760038+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:28.760038+0000","last_peered":"2021-01-15T18:41:28.760038+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:28.760038+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2456,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20062374,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1045}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2624","reported_seq":"2683","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.923308+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.923308+0000","last_peered":"2021-01-15T18:41:21.923308+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.923308+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2624,"ondisk_log_size":2624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21446822,"num_objects":2619,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2659","reported_seq":"2713","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.796505+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.796505+0000","last_peered":"2021-01-15T18:41:21.796505+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.796505+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2659,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21700940,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"30'2498","reported_seq":"2573","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.828886+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:28.828886+0000","last_peered":"2021-01-15T18:41:28.828886+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:28.828886+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2498,"ondisk_log_size":2498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20439206,"num_objects":2496,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1114}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329287956,"num_objects":40210,"num_object_clones":0,"num_object_copies":160840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40237,"num_write_kb":321686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":576299008,"data_stored":576299008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243100,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":82111,"kb_avail":94128740,"statfs":{"total":96636764160,"available":96387829760,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84081716},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.627},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239132,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":78144,"kb_avail":94132708,"statfs":{"total":96636764160,"available":96391892992,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80019456},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":6.3719999999999999},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":6.8650000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280539,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247260,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":86271,"kb_avail":94124580,"statfs":{"total":96636764160,"available":96383569920,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88341556},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.79300000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149772,"kb_used_data":80388,"kb_used_omap":0,"kb_used_meta":69375,"kb_avail":94222068,"statfs":{"total":96636764160,"available":96483397632,"internally_reserved":0,"allocated":82317312,"data_stored":82150219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":71040052},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2,18],"upper_bound":32768},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.55100000000000005}]}]}]}} 2021-01-15T18:41:37.900 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:38.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 98 20 40437 40417 3.22108 1.00781 0.0108031 0.0480472 2021-01-15T18:41:38.284 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:38.284 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20386/160840 objects degraded (12.675%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":148,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40210,"data_bytes":329287956,"bytes_used":900366336,"bytes_avail":385646690304,"bytes_total":386547056640,"degraded_objects":20386,"degraded_total":160840,"degraded_ratio":0.12674707784133299,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":652769,"recovering_keys_per_sec":0,"num_objects_recovered":799,"num_bytes_recovered":6529024,"num_keys_recovered":0,"write_bytes_sec":18837,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:38.297 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:38.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:38.600 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:38.602 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:38.613 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2021-01-15T18:41:38.391810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329664788,"num_objects":40256,"num_object_clones":0,"num_object_copies":161024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40283,"num_write_kb":322054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40310,"ondisk_log_size":40310,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":880224,"kb_used_data":563980,"kb_used_omap":2,"kb_used_meta":316221,"kb_avail":376607136,"statfs":{"total":386547056640,"available":385645707264,"internally_reserved":0,"allocated":577515520,"data_stored":576837879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":323810460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2,18],"upper_bound":32768},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":450560,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":6529024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001573"},"pg_stats":[{"pgid":"1.f","version":"30'2408","reported_seq":"2448","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.745771+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:28.745771+0000","last_peered":"2021-01-15T18:41:28.745771+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:28.745771+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2408,"ondisk_log_size":2408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19726336,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":19264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'2461","reported_seq":"5374","reported_epoch":"30","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:21.815654+0000","last_change":"2021-01-15T18:41:21.815654+0000","last_active":"2021-01-15T18:41:21.815654+0000","last_peered":"2021-01-15T18:41:21.815654+0000","last_clean":"2021-01-15T18:41:21.815654+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:21.815654+0000","last_undegraded":"2021-01-15T18:41:21.815654+0000","last_fullsized":"2021-01-15T18:41:21.815654+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20119718,"num_objects":2457,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2459,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"32'2523","reported_seq":"2598","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.497242+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:37.497242+0000","last_peered":"2021-01-15T18:41:37.497242+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:37.497242+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2523,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20639827,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"30'2561","reported_seq":"2601","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.849265+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:21.849265+0000","last_peered":"2021-01-15T18:41:21.849265+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:21.849265+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2561,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20979712,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"30'2578","reported_seq":"2636","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.860192+0000","last_change":"2021-01-15T18:40:32.756160+0000","last_active":"2021-01-15T18:41:21.860192+0000","last_peered":"2021-01-15T18:41:21.860192+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:21.860192+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2578,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21069990,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"4287","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:33.202640+0000","last_change":"2021-01-15T18:41:19.750336+0000","last_active":"2021-01-15T18:41:33.202640+0000","last_peered":"2021-01-15T18:41:33.202640+0000","last_clean":"2021-01-15T18:39:34.924332+0000","last_became_active":"2021-01-15T18:40:32.705786+0000","last_became_peered":"2021-01-15T18:40:32.705786+0000","last_unstale":"2021-01-15T18:41:33.202640+0000","last_undegraded":"2021-01-15T18:40:32.522654+0000","last_fullsized":"2021-01-15T18:40:32.487416+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":7061504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1999}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"30'2486","reported_seq":"2534","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.773579+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:28.773579+0000","last_peered":"2021-01-15T18:41:28.773579+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:28.773579+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2486,"ondisk_log_size":2486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1090}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2568","reported_seq":"2638","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.733938+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:37.733938+0000","last_peered":"2021-01-15T18:41:37.733938+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:37.733938+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2568,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21024851,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"30'2562","reported_seq":"2598","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.819311+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:28.819311+0000","last_peered":"2021-01-15T18:41:28.819311+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:28.819311+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2562,"ondisk_log_size":2562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20975699,"num_objects":2561,"num_object_clones":0,"num_object_copies":10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":20485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"30'2560","reported_seq":"2614","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.906913+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:21.906913+0000","last_peered":"2021-01-15T18:41:21.906913+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:21.906913+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2560,"ondisk_log_size":2560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20918521,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"30'2433","reported_seq":"2485","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.796392+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:28.796392+0000","last_peered":"2021-01-15T18:41:28.796392+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:28.796392+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19898534,"num_objects":2430,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'2433","reported_seq":"2491","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.782262+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:28.782262+0000","last_peered":"2021-01-15T18:41:28.782262+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:28.782262+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2433,"ondisk_log_size":2433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19890342,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"30'2456","reported_seq":"2516","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:28.760038+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:28.760038+0000","last_peered":"2021-01-15T18:41:28.760038+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:28.760038+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2456,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20062374,"num_objects":2450,"num_object_clones":0,"num_object_copies":9800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1045}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"30'2624","reported_seq":"2683","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.923308+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:21.923308+0000","last_peered":"2021-01-15T18:41:21.923308+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:21.923308+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2624,"ondisk_log_size":2624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21446822,"num_objects":2619,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"30'2659","reported_seq":"2713","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:21.796505+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:21.796505+0000","last_peered":"2021-01-15T18:41:21.796505+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:21.796505+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2659,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21700940,"num_objects":2651,"num_object_clones":0,"num_object_copies":10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2513","reported_seq":"2590","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.783022+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:37.783022+0000","last_peered":"2021-01-15T18:41:37.783022+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:37.783022+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20562086,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329664788,"num_objects":40256,"num_object_clones":0,"num_object_copies":161024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40283,"num_write_kb":322054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2311,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577261568,"data_stored":577261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40310,"ondisk_log_size":40310,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243100,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":82111,"kb_avail":94128740,"statfs":{"total":96636764160,"available":96387829760,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84081716},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.627},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240092,"kb_used_data":161632,"kb_used_omap":0,"kb_used_meta":78464,"kb_avail":94131748,"statfs":{"total":96636764160,"available":96390909952,"internally_reserved":0,"allocated":165511168,"data_stored":165334806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80347136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.434},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.69599999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280539,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247260,"kb_used_data":160980,"kb_used_omap":0,"kb_used_meta":86271,"kb_avail":94124580,"statfs":{"total":96636764160,"available":96383569920,"internally_reserved":0,"allocated":164843520,"data_stored":164676427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":88341556},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.79300000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149772,"kb_used_data":80388,"kb_used_omap":0,"kb_used_meta":69375,"kb_avail":94222068,"statfs":{"total":96636764160,"available":96483397632,"internally_reserved":0,"allocated":82317312,"data_stored":82150219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":71040052},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2,18],"upper_bound":32768},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.55100000000000005}]}]}]}} 2021-01-15T18:41:39.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 99 20 40437 40417 3.18855 0 - 0.0480472 2021-01-15T18:41:40.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:41:40.033387+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0480472 2021-01-15T18:41:40.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:41:40.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 100 20 40437 40417 3.15668 0 - 0.0480472 2021-01-15T18:41:41.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 101 20 40437 40417 3.12543 0 - 0.0480472 2021-01-15T18:41:41.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:41.928 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:41.930 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:41.943 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2021-01-15T18:41:40.392195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":15908864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":880944,"kb_used_data":569948,"kb_used_omap":2,"kb_used_meta":310973,"kb_avail":376606416,"statfs":{"total":386547056640,"available":385644969984,"internally_reserved":0,"allocated":583626752,"data_stored":582921304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":318436508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,12,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1998848,"num_objects":244,"num_object_clones":0,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":244,"num_write_kb":1952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":459,"num_bytes_recovered":3760128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001575"},"pg_stats":[{"pgid":"1.f","version":"32'2431","reported_seq":"2473","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.648190+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:37.648190+0000","last_peered":"2021-01-15T18:41:37.648190+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:37.648190+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19914752,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2497","reported_seq":"5413","reported_epoch":"32","state":"active+clean","last_fresh":"2021-01-15T18:41:37.541790+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:37.541790+0000","last_peered":"2021-01-15T18:41:37.541790+0000","last_clean":"2021-01-15T18:41:37.541790+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:37.541790+0000","last_undegraded":"2021-01-15T18:41:37.541790+0000","last_fullsized":"2021-01-15T18:41:37.541790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2497,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"32'2523","reported_seq":"2598","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.497242+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:37.497242+0000","last_peered":"2021-01-15T18:41:37.497242+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:37.497242+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2523,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20639827,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"32'2570","reported_seq":"2612","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.572796+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:37.572796+0000","last_peered":"2021-01-15T18:41:37.572796+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:37.572796+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2570,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3585","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:39.028797+0000","last_change":"2021-01-15T18:41:35.222811+0000","last_active":"2021-01-15T18:41:39.028797+0000","last_peered":"2021-01-15T18:41:39.028797+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:39.028797+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":3825664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1551}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2687","reported_epoch":"32","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:36.540067+0000","last_change":"2021-01-15T18:41:36.540067+0000","last_active":"2021-01-15T18:41:36.540067+0000","last_peered":"2021-01-15T18:41:36.540067+0000","last_clean":"2021-01-15T18:41:36.540067+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:36.540067+0000","last_undegraded":"2021-01-15T18:41:36.540067+0000","last_fullsized":"2021-01-15T18:41:36.540067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"32'2504","reported_seq":"2554","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.574053+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:37.574053+0000","last_peered":"2021-01-15T18:41:37.574053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:37.574053+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2504,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2568","reported_seq":"2638","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.733938+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:37.733938+0000","last_peered":"2021-01-15T18:41:37.733938+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:37.733938+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2568,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21024851,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"32'2575","reported_seq":"2613","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.553140+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:37.553140+0000","last_peered":"2021-01-15T18:41:37.553140+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:37.553140+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2575,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21082195,"num_objects":2574,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1138}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"32'2569","reported_seq":"2625","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.445591+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:37.445591+0000","last_peered":"2021-01-15T18:41:37.445591+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:37.445591+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2569,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20992249,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"32'2441","reported_seq":"2495","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.593377+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:37.593377+0000","last_peered":"2021-01-15T18:41:37.593377+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:37.593377+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2441,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"32'2450","reported_seq":"2510","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.702808+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:37.702808+0000","last_peered":"2021-01-15T18:41:37.702808+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:37.702808+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2450,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":19567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"32'2466","reported_seq":"2528","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.234228+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:37.234228+0000","last_peered":"2021-01-15T18:41:37.234228+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:37.234228+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2466,"ondisk_log_size":2466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20144294,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2639","reported_seq":"2700","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.432663+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:37.432663+0000","last_peered":"2021-01-15T18:41:37.432663+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:37.432663+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2639,"ondisk_log_size":2639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569702,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2678","reported_seq":"2734","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.679903+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:37.679903+0000","last_peered":"2021-01-15T18:41:37.679903+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:37.679903+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2678,"ondisk_log_size":2678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856588,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2513","reported_seq":"2590","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.783022+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:37.783022+0000","last_peered":"2021-01-15T18:41:37.783022+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:37.783022+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20562086,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":15908864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":583192576,"data_stored":583192576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244568,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":82623,"kb_avail":94127272,"statfs":{"total":96636764160,"available":96386326528,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84606004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.999}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240092,"kb_used_data":161632,"kb_used_omap":0,"kb_used_meta":78464,"kb_avail":94131748,"statfs":{"total":96636764160,"available":96390909952,"internally_reserved":0,"allocated":165511168,"data_stored":165334806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80347136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.434},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.69599999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252760,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":90815,"kb_avail":94119080,"statfs":{"total":96636764160,"available":96377937920,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92994612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.7869999999999999},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.65400000000000003},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.25}]}]},{"osd":1,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143524,"kb_used_data":84444,"kb_used_omap":0,"kb_used_meta":59071,"kb_avail":94228316,"statfs":{"total":96636764160,"available":96489795584,"internally_reserved":0,"allocated":86470656,"data_stored":86294294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60488756},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,12,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.505},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.79800000000000004}]}]}]}} 2021-01-15T18:41:41.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:42.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 102 20 40437 40417 3.09479 0 - 0.0480472 2021-01-15T18:41:42.326 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:42.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19434/161780 objects degraded (12.013%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":152,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40445,"data_bytes":331213076,"bytes_used":902086656,"bytes_avail":385644969984,"bytes_total":386547056640,"degraded_objects":19434,"degraded_total":161780,"degraded_ratio":0.12012609716899494,"recovering_objects_per_sec":57,"recovering_bytes_per_sec":469923,"recovering_keys_per_sec":0,"num_objects_recovered":459,"num_bytes_recovered":3760128,"num_keys_recovered":0,"write_bytes_sec":249806,"read_op_per_sec":0,"write_op_per_sec":30},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:42.340 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:42.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:42.652 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:42.654 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:42.666 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2021-01-15T18:41:42.392607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":15908864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":880944,"kb_used_data":569948,"kb_used_omap":2,"kb_used_meta":310973,"kb_avail":376606416,"statfs":{"total":386547056640,"available":385644969984,"internally_reserved":0,"allocated":583626752,"data_stored":582921304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":318436508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,12,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1925120,"num_objects":235,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":1880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-340,"num_bytes_recovered":-2768896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001596"},"pg_stats":[{"pgid":"1.f","version":"32'2431","reported_seq":"2473","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.648190+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:37.648190+0000","last_peered":"2021-01-15T18:41:37.648190+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:37.648190+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19914752,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2497","reported_seq":"5413","reported_epoch":"32","state":"active+clean","last_fresh":"2021-01-15T18:41:37.541790+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:37.541790+0000","last_peered":"2021-01-15T18:41:37.541790+0000","last_clean":"2021-01-15T18:41:37.541790+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:37.541790+0000","last_undegraded":"2021-01-15T18:41:37.541790+0000","last_fullsized":"2021-01-15T18:41:37.541790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2497,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"32'2523","reported_seq":"2598","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.497242+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:37.497242+0000","last_peered":"2021-01-15T18:41:37.497242+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:37.497242+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2523,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20639827,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"32'2570","reported_seq":"2612","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.572796+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:37.572796+0000","last_peered":"2021-01-15T18:41:37.572796+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:37.572796+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2570,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3585","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:39.028797+0000","last_change":"2021-01-15T18:41:35.222811+0000","last_active":"2021-01-15T18:41:39.028797+0000","last_peered":"2021-01-15T18:41:39.028797+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:39.028797+0000","last_undegraded":"2021-01-15T18:40:32.504276+0000","last_fullsized":"2021-01-15T18:40:32.487285+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":3825664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1551}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2687","reported_epoch":"32","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:36.540067+0000","last_change":"2021-01-15T18:41:36.540067+0000","last_active":"2021-01-15T18:41:36.540067+0000","last_peered":"2021-01-15T18:41:36.540067+0000","last_clean":"2021-01-15T18:41:36.540067+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:36.540067+0000","last_undegraded":"2021-01-15T18:41:36.540067+0000","last_fullsized":"2021-01-15T18:41:36.540067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"32'2504","reported_seq":"2554","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.574053+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:37.574053+0000","last_peered":"2021-01-15T18:41:37.574053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:37.574053+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2504,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2568","reported_seq":"2638","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.733938+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:37.733938+0000","last_peered":"2021-01-15T18:41:37.733938+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:37.733938+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2568,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21024851,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"32'2575","reported_seq":"2613","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.553140+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:37.553140+0000","last_peered":"2021-01-15T18:41:37.553140+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:37.553140+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2575,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21082195,"num_objects":2574,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1138}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"32'2569","reported_seq":"2625","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.445591+0000","last_change":"2021-01-15T18:40:32.737008+0000","last_active":"2021-01-15T18:41:37.445591+0000","last_peered":"2021-01-15T18:41:37.445591+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:37.445591+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2569,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20992249,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"32'2441","reported_seq":"2495","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.593377+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:37.593377+0000","last_peered":"2021-01-15T18:41:37.593377+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:37.593377+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2441,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"32'2450","reported_seq":"2510","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.702808+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:37.702808+0000","last_peered":"2021-01-15T18:41:37.702808+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:37.702808+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2450,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":19567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"32'2466","reported_seq":"2528","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.234228+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:37.234228+0000","last_peered":"2021-01-15T18:41:37.234228+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:37.234228+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2466,"ondisk_log_size":2466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20144294,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2639","reported_seq":"2700","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.432663+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:37.432663+0000","last_peered":"2021-01-15T18:41:37.432663+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:37.432663+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2639,"ondisk_log_size":2639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569702,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2678","reported_seq":"2734","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.679903+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:37.679903+0000","last_peered":"2021-01-15T18:41:37.679903+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:37.679903+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2678,"ondisk_log_size":2678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856588,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2513","reported_seq":"2590","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.783022+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:37.783022+0000","last_peered":"2021-01-15T18:41:37.783022+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:37.783022+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20562086,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1971,"num_bytes_recovered":15908864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":583192576,"data_stored":583192576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244568,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":82623,"kb_avail":94127272,"statfs":{"total":96636764160,"available":96386326528,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84606004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.999}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240092,"kb_used_data":161632,"kb_used_omap":0,"kb_used_meta":78464,"kb_avail":94131748,"statfs":{"total":96636764160,"available":96390909952,"internally_reserved":0,"allocated":165511168,"data_stored":165334806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80347136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.434},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.69599999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252760,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":90815,"kb_avail":94119080,"statfs":{"total":96636764160,"available":96377937920,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92994612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.7869999999999999},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.65400000000000003},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":1.25}]}]},{"osd":1,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143524,"kb_used_data":84444,"kb_used_omap":0,"kb_used_meta":59071,"kb_avail":94228316,"statfs":{"total":96636764160,"available":96489795584,"internally_reserved":0,"allocated":86470656,"data_stored":86294294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60488756},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,12,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.505},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.79800000000000004}]}]}]}} 2021-01-15T18:41:43.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 103 20 40437 40417 3.06475 0 - 0.0480472 2021-01-15T18:41:44.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 104 20 40437 40417 3.03529 0 - 0.0480472 2021-01-15T18:41:45.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 105 20 40527 40507 3.01308 0.100446 0.0252841 0.0518136 2021-01-15T18:41:45.668 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:46.035 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 106 20 40751 40731 3.00116 1.75 0.0105104 0.0516865 2021-01-15T18:41:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:46.179 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2021-01-15T18:41:45.500858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":895816,"kb_used_data":574440,"kb_used_omap":2,"kb_used_meta":321341,"kb_avail":376591544,"statfs":{"total":386547056640,"available":385629741056,"internally_reserved":0,"allocated":588226560,"data_stored":587521112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":329053340},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1925120,"num_objects":235,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":1880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-494,"num_bytes_recovered":-4030464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.984660"},"pg_stats":[{"pgid":"1.f","version":"32'2431","reported_seq":"2473","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.648190+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:37.648190+0000","last_peered":"2021-01-15T18:41:37.648190+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:37.648190+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19914752,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2497","reported_seq":"5414","reported_epoch":"32","state":"active+clean","last_fresh":"2021-01-15T18:41:37.724836+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:37.724836+0000","last_peered":"2021-01-15T18:41:37.724836+0000","last_clean":"2021-01-15T18:41:37.724836+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:37.724836+0000","last_undegraded":"2021-01-15T18:41:37.724836+0000","last_fullsized":"2021-01-15T18:41:37.724836+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2497,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"32'2523","reported_seq":"2598","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.497242+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:37.497242+0000","last_peered":"2021-01-15T18:41:37.497242+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:37.497242+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2523,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20639827,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"32'2570","reported_seq":"2612","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.572796+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:37.572796+0000","last_peered":"2021-01-15T18:41:37.572796+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:37.572796+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2570,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3165","reported_epoch":"33","state":"peering","last_fresh":"2021-01-15T18:41:44.493924+0000","last_change":"2021-01-15T18:41:44.493732+0000","last_active":"2021-01-15T18:41:44.493464+0000","last_peered":"2021-01-15T18:41:37.322280+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:44.493924+0000","last_undegraded":"2021-01-15T18:41:44.493924+0000","last_fullsized":"2021-01-15T18:41:44.493924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2688","reported_epoch":"32","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:36.542516+0000","last_change":"2021-01-15T18:41:36.540067+0000","last_active":"2021-01-15T18:41:36.542516+0000","last_peered":"2021-01-15T18:41:36.542516+0000","last_clean":"2021-01-15T18:41:36.542516+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:36.542516+0000","last_undegraded":"2021-01-15T18:41:36.542516+0000","last_fullsized":"2021-01-15T18:41:36.542516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"32'2504","reported_seq":"2554","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.574053+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:37.574053+0000","last_peered":"2021-01-15T18:41:37.574053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:37.574053+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2504,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2568","reported_seq":"2638","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.733938+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:37.733938+0000","last_peered":"2021-01-15T18:41:37.733938+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:37.733938+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2568,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21024851,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"32'2575","reported_seq":"2613","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.553140+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:37.553140+0000","last_peered":"2021-01-15T18:41:37.553140+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:37.553140+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2575,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21082195,"num_objects":2574,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1138}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"32'2569","reported_seq":"2742","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:44.314292+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:44.314292+0000","last_peered":"2021-01-15T18:41:44.314292+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:44.314292+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2569,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20992249,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":475136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"32'2441","reported_seq":"2495","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.593377+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:37.593377+0000","last_peered":"2021-01-15T18:41:37.593377+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:37.593377+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2441,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"32'2450","reported_seq":"2510","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.702808+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:37.702808+0000","last_peered":"2021-01-15T18:41:37.702808+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:37.702808+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2450,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":19567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"32'2466","reported_seq":"2528","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.234228+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:37.234228+0000","last_peered":"2021-01-15T18:41:37.234228+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:37.234228+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2466,"ondisk_log_size":2466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20144294,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2639","reported_seq":"2700","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.432663+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:37.432663+0000","last_peered":"2021-01-15T18:41:37.432663+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:37.432663+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2639,"ondisk_log_size":2639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569702,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2678","reported_seq":"2734","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.679903+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:37.679903+0000","last_peered":"2021-01-15T18:41:37.679903+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:37.679903+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2678,"ondisk_log_size":2678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856588,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2513","reported_seq":"2590","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.783022+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:37.783022+0000","last_peered":"2021-01-15T18:41:37.783022+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:37.783022+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20562086,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":587415552,"data_stored":587415552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244568,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":82623,"kb_avail":94127272,"statfs":{"total":96636764160,"available":96386326528,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84606004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240600,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":78656,"kb_avail":94131240,"statfs":{"total":96636764160,"available":96390389760,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80543744},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.54400000000000004},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.61499999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252760,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":90815,"kb_avail":94119080,"statfs":{"total":96636764160,"available":96377937920,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92994612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157888,"kb_used_data":88632,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94213952,"statfs":{"total":96636764160,"available":96475086848,"internally_reserved":0,"allocated":90759168,"data_stored":90582806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.495}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:46.181 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:46.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:46.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18339/161780 objects degraded (11.336%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":157,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1},{"state_name":"active+clean+wait","count":1},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40445,"data_bytes":331213076,"bytes_used":917315584,"bytes_avail":385629741056,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":18339,"degraded_total":161780,"degraded_ratio":0.11335764618617876,"write_bytes_sec":193497,"read_op_per_sec":0,"write_op_per_sec":23},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:46.592 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2021-01-15T18:41:46.593 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:46.899 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:46.901 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:46.913 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2021-01-15T18:41:46.393413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":895816,"kb_used_data":574440,"kb_used_omap":2,"kb_used_meta":321341,"kb_avail":376591544,"statfs":{"total":386547056640,"available":385629741056,"internally_reserved":0,"allocated":588226560,"data_stored":587521112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":329053340},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1548288,"num_objects":189,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":189,"num_write_kb":1512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-494,"num_bytes_recovered":-4030464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001603"},"pg_stats":[{"pgid":"1.f","version":"32'2431","reported_seq":"2473","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.648190+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:37.648190+0000","last_peered":"2021-01-15T18:41:37.648190+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:37.648190+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19914752,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2497","reported_seq":"5414","reported_epoch":"32","state":"active+clean","last_fresh":"2021-01-15T18:41:37.724836+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:37.724836+0000","last_peered":"2021-01-15T18:41:37.724836+0000","last_clean":"2021-01-15T18:41:37.724836+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:37.724836+0000","last_undegraded":"2021-01-15T18:41:37.724836+0000","last_fullsized":"2021-01-15T18:41:37.724836+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2497,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"32'2523","reported_seq":"2598","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.497242+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:37.497242+0000","last_peered":"2021-01-15T18:41:37.497242+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:37.497242+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2523,"ondisk_log_size":2523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20639827,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":20162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"32'2570","reported_seq":"2612","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.572796+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:37.572796+0000","last_peered":"2021-01-15T18:41:37.572796+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:37.572796+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2570,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3165","reported_epoch":"33","state":"peering","last_fresh":"2021-01-15T18:41:44.493924+0000","last_change":"2021-01-15T18:41:44.493732+0000","last_active":"2021-01-15T18:41:44.493464+0000","last_peered":"2021-01-15T18:41:37.322280+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:44.493924+0000","last_undegraded":"2021-01-15T18:41:44.493924+0000","last_fullsized":"2021-01-15T18:41:44.493924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2688","reported_epoch":"32","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:36.542516+0000","last_change":"2021-01-15T18:41:36.540067+0000","last_active":"2021-01-15T18:41:36.542516+0000","last_peered":"2021-01-15T18:41:36.542516+0000","last_clean":"2021-01-15T18:41:36.542516+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:36.542516+0000","last_undegraded":"2021-01-15T18:41:36.542516+0000","last_fullsized":"2021-01-15T18:41:36.542516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"32'2504","reported_seq":"2554","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.574053+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:37.574053+0000","last_peered":"2021-01-15T18:41:37.574053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:37.574053+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2504,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"32'2568","reported_seq":"2638","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.733938+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:37.733938+0000","last_peered":"2021-01-15T18:41:37.733938+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:37.733938+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2568,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21024851,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"32'2575","reported_seq":"2613","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.553140+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:37.553140+0000","last_peered":"2021-01-15T18:41:37.553140+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:37.553140+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2575,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21082195,"num_objects":2574,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1138}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"32'2569","reported_seq":"2742","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:44.314292+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:44.314292+0000","last_peered":"2021-01-15T18:41:44.314292+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:44.314292+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2569,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20992249,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":475136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"32'2441","reported_seq":"2495","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.593377+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:37.593377+0000","last_peered":"2021-01-15T18:41:37.593377+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:37.593377+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2441,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"32'2450","reported_seq":"2510","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.702808+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:37.702808+0000","last_peered":"2021-01-15T18:41:37.702808+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:37.702808+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2450,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":19567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"32'2466","reported_seq":"2528","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.234228+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:37.234228+0000","last_peered":"2021-01-15T18:41:37.234228+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:37.234228+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2466,"ondisk_log_size":2466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20144294,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2639","reported_seq":"2700","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.432663+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:37.432663+0000","last_peered":"2021-01-15T18:41:37.432663+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:37.432663+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2639,"ondisk_log_size":2639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569702,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2678","reported_seq":"2734","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.679903+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:37.679903+0000","last_peered":"2021-01-15T18:41:37.679903+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:37.679903+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2678,"ondisk_log_size":2678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856588,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"32'2513","reported_seq":"2590","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.783022+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:37.783022+0000","last_peered":"2021-01-15T18:41:37.783022+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:37.783022+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2513,"ondisk_log_size":2513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20562086,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":20082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331213076,"num_objects":40445,"num_object_clones":0,"num_object_copies":161780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40472,"num_write_kb":323566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":587415552,"data_stored":587415552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40499,"ondisk_log_size":40499,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244568,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":82623,"kb_avail":94127272,"statfs":{"total":96636764160,"available":96386326528,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84606004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240600,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":78656,"kb_avail":94131240,"statfs":{"total":96636764160,"available":96390389760,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80543744},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:44 2021","interfaces":[{"interface":"back","average":{"1min":3.7549999999999999,"5min":3.7549999999999999,"15min":3.7549999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":36.380000000000003,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.54400000000000004},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.61499999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252760,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":90815,"kb_avail":94119080,"statfs":{"total":96636764160,"available":96377937920,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92994612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157888,"kb_used_data":88632,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94213952,"statfs":{"total":96636764160,"available":96475086848,"internally_reserved":0,"allocated":90759168,"data_stored":90582806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.495}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:47.033 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 107 20 40780 40760 2.97524 0.226562 0.0137269 0.0516569 2021-01-15T18:41:48.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 108 20 40780 40760 2.9477 0 - 0.0516569 2021-01-15T18:41:49.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 109 20 40780 40760 2.92066 0 - 0.0516569 2021-01-15T18:41:49.916 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:50.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 110 20 40780 40760 2.89411 0 - 0.0516569 2021-01-15T18:41:50.253 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:50.266 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2021-01-15T18:41:48.393935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331647252,"num_objects":40498,"num_object_clones":0,"num_object_copies":161992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40525,"num_write_kb":323990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40552,"ondisk_log_size":40552,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":902004,"kb_used_data":575828,"kb_used_omap":2,"kb_used_meta":326141,"kb_avail":376585356,"statfs":{"total":386547056640,"available":385623404544,"internally_reserved":0,"allocated":589647872,"data_stored":588933081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":333968540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":434176,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-154,"num_bytes_recovered":-1261568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001740"},"pg_stats":[{"pgid":"1.f","version":"32'2431","reported_seq":"2473","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.648190+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:37.648190+0000","last_peered":"2021-01-15T18:41:37.648190+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:37.648190+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2431,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19914752,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":19448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'2497","reported_seq":"5414","reported_epoch":"32","state":"active+clean","last_fresh":"2021-01-15T18:41:37.724836+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:37.724836+0000","last_peered":"2021-01-15T18:41:37.724836+0000","last_clean":"2021-01-15T18:41:37.724836+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:37.724836+0000","last_undegraded":"2021-01-15T18:41:37.724836+0000","last_fullsized":"2021-01-15T18:41:37.724836+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2497,"ondisk_log_size":2497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20414630,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2545","reported_seq":"2622","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.003031+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:46.003031+0000","last_peered":"2021-01-15T18:41:46.003031+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:46.003031+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2545,"ondisk_log_size":2545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20820051,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2544,"num_write_kb":20338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"32'2570","reported_seq":"2612","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.572796+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:37.572796+0000","last_peered":"2021-01-15T18:41:37.572796+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:37.572796+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2570,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21053440,"num_objects":2570,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3165","reported_epoch":"33","state":"peering","last_fresh":"2021-01-15T18:41:44.493924+0000","last_change":"2021-01-15T18:41:44.493732+0000","last_active":"2021-01-15T18:41:44.493464+0000","last_peered":"2021-01-15T18:41:37.322280+0000","last_clean":"2021-01-15T18:39:33.500147+0000","last_became_active":"2021-01-15T18:40:32.750719+0000","last_became_peered":"2021-01-15T18:40:32.750719+0000","last_unstale":"2021-01-15T18:41:44.493924+0000","last_undegraded":"2021-01-15T18:41:44.493924+0000","last_fullsized":"2021-01-15T18:41:44.493924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"30'2485","reported_seq":"2688","reported_epoch":"32","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:36.542516+0000","last_change":"2021-01-15T18:41:36.540067+0000","last_active":"2021-01-15T18:41:36.542516+0000","last_peered":"2021-01-15T18:41:36.542516+0000","last_clean":"2021-01-15T18:41:36.542516+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:36.542516+0000","last_undegraded":"2021-01-15T18:41:36.542516+0000","last_fullsized":"2021-01-15T18:41:36.542516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2485,"ondisk_log_size":2485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":19855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"32'2504","reported_seq":"2554","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.574053+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:37.574053+0000","last_peered":"2021-01-15T18:41:37.574053+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:37.574053+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2504,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1108}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2582","reported_seq":"2654","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.509677+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:45.509677+0000","last_peered":"2021-01-15T18:41:45.509677+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:45.509677+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2582,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21139539,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"32'2575","reported_seq":"2613","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.553140+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:37.553140+0000","last_peered":"2021-01-15T18:41:37.553140+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:37.553140+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2575,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21082195,"num_objects":2574,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1138}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"32'2569","reported_seq":"2742","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:44.314292+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:44.314292+0000","last_peered":"2021-01-15T18:41:44.314292+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:44.314292+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2569,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20992249,"num_objects":2564,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":20508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":475136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"32'2441","reported_seq":"2495","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.593377+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:37.593377+0000","last_peered":"2021-01-15T18:41:37.593377+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:37.593377+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2441,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1057}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"32'2450","reported_seq":"2510","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.702808+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:37.702808+0000","last_peered":"2021-01-15T18:41:37.702808+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:37.702808+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2450,"ondisk_log_size":2450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20029606,"num_objects":2446,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":19567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"32'2466","reported_seq":"2528","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.234228+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:37.234228+0000","last_peered":"2021-01-15T18:41:37.234228+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:37.234228+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2466,"ondisk_log_size":2466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20144294,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1055}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'2639","reported_seq":"2700","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.432663+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:37.432663+0000","last_peered":"2021-01-15T18:41:37.432663+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:37.432663+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2639,"ondisk_log_size":2639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569702,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1192}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"32'2678","reported_seq":"2734","reported_epoch":"32","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:37.679903+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:37.679903+0000","last_peered":"2021-01-15T18:41:37.679903+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:37.679903+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2678,"ondisk_log_size":2678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856588,"num_objects":2670,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2530","reported_seq":"2609","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.144769+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:46.144769+0000","last_peered":"2021-01-15T18:41:46.144769+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:46.144769+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2530,"ondisk_log_size":2530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20701350,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":20218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331647252,"num_objects":40498,"num_object_clones":0,"num_object_copies":161992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40525,"num_write_kb":323990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1817,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":588820480,"data_stored":588820480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40552,"ondisk_log_size":40552,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244568,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":82623,"kb_avail":94127272,"statfs":{"total":96636764160,"available":96386326528,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84606004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.59199999999999997},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.999}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246788,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":83456,"kb_avail":94125052,"statfs":{"total":96636764160,"available":96384053248,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":85458944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.0329999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.64300000000000002},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.70999999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280541,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252760,"kb_used_data":161936,"kb_used_omap":0,"kb_used_meta":90815,"kb_avail":94119080,"statfs":{"total":96636764160,"available":96377937920,"internally_reserved":0,"allocated":165822464,"data_stored":165646102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92994612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:40:45 2021","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.705,"5min":13.705,"15min":13.705},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":13.238,"5min":13.238,"15min":13.238},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157888,"kb_used_data":88632,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94213952,"statfs":{"total":96636764160,"available":96475086848,"internally_reserved":0,"allocated":90759168,"data_stored":90582806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.495}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.98199999999999998}]}]}]}} 2021-01-15T18:41:50.268 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:50.645 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:50.646 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17720/163152 objects degraded (10.861%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40788,"data_bytes":334022932,"bytes_used":917229568,"bytes_avail":385629827072,"bytes_total":386547056640,"degraded_objects":17720,"degraded_total":163152,"degraded_ratio":0.10861037560066686,"recovering_objects_per_sec":214,"recovering_bytes_per_sec":1751686,"recovering_keys_per_sec":0,"num_objects_recovered":1713,"num_bytes_recovered":14016512,"num_keys_recovered":0,"write_bytes_sec":351156,"read_op_per_sec":0,"write_op_per_sec":42},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:50.659 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:50.660 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:50.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:50.984 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:50.996 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2021-01-15T18:41:50.394331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334022932,"num_objects":40788,"num_object_clones":0,"num_object_copies":163152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40815,"num_write_kb":326310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3684,"num_bytes_recovered":29925376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40842,"ondisk_log_size":40842,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":895732,"kb_used_data":582548,"kb_used_omap":2,"kb_used_meta":313149,"kb_avail":376591628,"statfs":{"total":386547056640,"available":385629827072,"internally_reserved":0,"allocated":596529152,"data_stored":595786332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":320664732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2809856,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":343,"num_write_kb":2744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1713,"num_bytes_recovered":14016512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001724"},"pg_stats":[{"pgid":"1.f","version":"34'2453","reported_seq":"2497","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.888877+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:45.888877+0000","last_peered":"2021-01-15T18:41:45.888877+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:45.888877+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2453,"ondisk_log_size":2453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20094976,"num_objects":2453,"num_object_clones":0,"num_object_copies":9812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":19624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'2518","reported_seq":"5436","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:46.137258+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:46.137258+0000","last_peered":"2021-01-15T18:41:46.137258+0000","last_clean":"2021-01-15T18:41:46.137258+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:46.137258+0000","last_undegraded":"2021-01-15T18:41:46.137258+0000","last_fullsized":"2021-01-15T18:41:46.137258+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2518,"ondisk_log_size":2518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20586662,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2545","reported_seq":"2622","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.003031+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:46.003031+0000","last_peered":"2021-01-15T18:41:46.003031+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:46.003031+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2545,"ondisk_log_size":2545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20820051,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2544,"num_write_kb":20338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"34'2599","reported_seq":"2643","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.257673+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:46.257673+0000","last_peered":"2021-01-15T18:41:46.257673+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:46.257673+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2599,"ondisk_log_size":2599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21291008,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":20792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3175","reported_epoch":"34","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:45.561887+0000","last_change":"2021-01-15T18:41:45.561887+0000","last_active":"2021-01-15T18:41:45.561887+0000","last_peered":"2021-01-15T18:41:45.561887+0000","last_clean":"2021-01-15T18:41:45.561887+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:41:45.561887+0000","last_undegraded":"2021-01-15T18:41:45.561887+0000","last_fullsized":"2021-01-15T18:41:45.561887+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"34'2531","reported_seq":"2736","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:46.023053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:41:46.023053+0000","last_peered":"2021-01-15T18:41:46.023053+0000","last_clean":"2021-01-15T18:41:46.023053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:46.023053+0000","last_undegraded":"2021-01-15T18:41:46.023053+0000","last_fullsized":"2021-01-15T18:41:46.023053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2531,"ondisk_log_size":2531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20701350,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":20223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"34'2520","reported_seq":"2572","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.973848+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:45.973848+0000","last_peered":"2021-01-15T18:41:45.973848+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:45.973848+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2520,"ondisk_log_size":2520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20603046,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2518,"num_write_kb":20132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2582","reported_seq":"2654","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.509677+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:45.509677+0000","last_peered":"2021-01-15T18:41:45.509677+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:45.509677+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2582,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21139539,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"34'2597","reported_seq":"2637","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.238906+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:46.238906+0000","last_peered":"2021-01-15T18:41:46.238906+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:46.238906+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2597,"ondisk_log_size":2597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21262419,"num_objects":2596,"num_object_clones":0,"num_object_copies":10384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2596,"num_write_kb":20765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1160}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"34'2587","reported_seq":"4000","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:49.316593+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:49.316593+0000","last_peered":"2021-01-15T18:41:49.316593+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:49.316593+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2587,"ondisk_log_size":2587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21139705,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":20652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":5545984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"34'2467","reported_seq":"2523","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.957622+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:45.957622+0000","last_peered":"2021-01-15T18:41:45.957622+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:45.957622+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2467,"ondisk_log_size":2467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20177062,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":19711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'2475","reported_seq":"2537","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.266404+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:46.266404+0000","last_peered":"2021-01-15T18:41:46.266404+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:46.266404+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2475,"ondisk_log_size":2475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20234406,"num_objects":2471,"num_object_clones":0,"num_object_copies":9884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":19767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"34'2492","reported_seq":"2556","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.936692+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:45.936692+0000","last_peered":"2021-01-15T18:41:45.936692+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:45.936692+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2492,"ondisk_log_size":2492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20357286,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1081}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'2659","reported_seq":"2722","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:45.892955+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:45.892955+0000","last_peered":"2021-01-15T18:41:45.892955+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:45.892955+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2659,"ondisk_log_size":2659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21733542,"num_objects":2654,"num_object_clones":0,"num_object_copies":10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":21236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'2697","reported_seq":"2755","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.217928+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:46.217928+0000","last_peered":"2021-01-15T18:41:46.217928+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:46.217928+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2697,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22012236,"num_objects":2689,"num_object_clones":0,"num_object_copies":10756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2694,"num_write_kb":21515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1203}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2530","reported_seq":"2609","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:46.144769+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:46.144769+0000","last_peered":"2021-01-15T18:41:46.144769+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:46.144769+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2530,"ondisk_log_size":2530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20701350,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":20218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":334022932,"num_objects":40788,"num_object_clones":0,"num_object_copies":163152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40815,"num_write_kb":326310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3684,"num_bytes_recovered":29925376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595488768,"data_stored":595488768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40842,"ondisk_log_size":40842,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215132,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250756,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":87423,"kb_avail":94121084,"statfs":{"total":96636764160,"available":96379990016,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":89521204},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":3.0089999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":14.565,"5min":14.565,"15min":14.565},"last":0.36599999999999999},{"interface":"front","average":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":14.435,"5min":14.435,"15min":14.435},"last":0.63300000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:40:50 2021","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3849999999999998,"5min":8.3849999999999998,"15min":8.3849999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.847,"5min":2.847,"15min":2.847},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":12.459,"5min":12.459,"15min":12.459},"last":0.56799999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246788,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":83456,"kb_avail":94125052,"statfs":{"total":96636764160,"available":96384053248,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":85458944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.0329999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:40:51 2021","interfaces":[{"interface":"back","average":{"1min":2.29,"5min":2.29,"15min":2.29},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.64300000000000002},{"interface":"front","average":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.4849999999999994,"5min":9.4849999999999994,"15min":9.4849999999999994},"last":0.70999999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":242628,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":79295,"kb_avail":94129212,"statfs":{"total":96636764160,"available":96388313088,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81198132},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.66700000000000004},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.502},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.68899999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155560,"kb_used_data":92576,"kb_used_omap":0,"kb_used_meta":62975,"kb_avail":94216280,"statfs":{"total":96636764160,"available":96477470720,"internally_reserved":0,"allocated":94797824,"data_stored":94612119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":64486452},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.503}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.65600000000000003}]}]}]}} 2021-01-15T18:41:51.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 111 20 40780 40760 2.86804 0 - 0.0516569 2021-01-15T18:41:52.034 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 112 20 40780 40760 2.84244 0 - 0.0516569 2021-01-15T18:41:53.036 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 113 20 40867 40847 2.82325 0.113281 0.0531495 0.0553266 2021-01-15T18:41:53.998 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:54.038 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 114 20 41248 41228 2.82454 2.97656 0.0400765 0.0552992 2021-01-15T18:41:55.039 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 115 20 41600 41580 2.82386 2.75 0.0565278 0.0553086 2021-01-15T18:41:55.155 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:55.183 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:55.208 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2021-01-15T18:41:54.398139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":336873748,"num_objects":41136,"num_object_clones":0,"num_object_copies":164544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41163,"num_write_kb":329094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4310,"num_bytes_recovered":35028992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41190,"ondisk_log_size":41190,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":902960,"kb_used_data":584808,"kb_used_omap":2,"kb_used_meta":318141,"kb_avail":376584400,"statfs":{"total":386547056640,"available":385622425600,"internally_reserved":0,"allocated":598843392,"data_stored":598100572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":325776540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5660672,"num_objects":691,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":691,"num_write_kb":5528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1245,"num_bytes_recovered":10174464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.909252"},"pg_stats":[{"pgid":"1.f","version":"34'2490","reported_seq":"2534","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.262303+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:54.262303+0000","last_peered":"2021-01-15T18:41:54.262303+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:54.262303+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2490,"ondisk_log_size":2490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20398080,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'2518","reported_seq":"5436","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:46.137258+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:46.137258+0000","last_peered":"2021-01-15T18:41:46.137258+0000","last_clean":"2021-01-15T18:41:46.137258+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:46.137258+0000","last_undegraded":"2021-01-15T18:41:46.137258+0000","last_fullsized":"2021-01-15T18:41:46.137258+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2518,"ondisk_log_size":2518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20586662,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2554","reported_seq":"2631","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.191245+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:53.191245+0000","last_peered":"2021-01-15T18:41:53.191245+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:53.191245+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2554,"ondisk_log_size":2554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20893779,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":20410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1135}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"34'2625","reported_seq":"2669","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.222694+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:54.222694+0000","last_peered":"2021-01-15T18:41:54.222694+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:54.222694+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2625,"ondisk_log_size":2625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21504000,"num_objects":2625,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":21000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"32'2590","reported_seq":"3175","reported_epoch":"34","state":"active+clean+wait","last_fresh":"2021-01-15T18:41:45.561887+0000","last_change":"2021-01-15T18:41:45.561887+0000","last_active":"2021-01-15T18:41:45.561887+0000","last_peered":"2021-01-15T18:41:45.561887+0000","last_clean":"2021-01-15T18:41:45.561887+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:41:45.561887+0000","last_undegraded":"2021-01-15T18:41:45.561887+0000","last_fullsized":"2021-01-15T18:41:45.561887+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2590,"ondisk_log_size":2590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21168294,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":20684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"34'2531","reported_seq":"2736","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:46.023053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:41:46.023053+0000","last_peered":"2021-01-15T18:41:46.023053+0000","last_clean":"2021-01-15T18:41:46.023053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:46.023053+0000","last_undegraded":"2021-01-15T18:41:46.023053+0000","last_fullsized":"2021-01-15T18:41:46.023053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2531,"ondisk_log_size":2531,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20701350,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":20223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"34'2559","reported_seq":"2611","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.200003+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:54.200003+0000","last_peered":"2021-01-15T18:41:54.200003+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:54.200003+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20922534,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1163}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2588","reported_seq":"2659","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.029390+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:53.029390+0000","last_peered":"2021-01-15T18:41:53.029390+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:53.029390+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2588,"ondisk_log_size":2588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21188691,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":20693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"34'2635","reported_seq":"2675","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.285804+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:54.285804+0000","last_peered":"2021-01-15T18:41:54.285804+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:54.285804+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2635,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21573715,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":21069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"34'2634","reported_seq":"5299","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.300982+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:54.300982+0000","last_peered":"2021-01-15T18:41:54.300982+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:54.300982+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2634,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21524729,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1306,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"34'2503","reported_seq":"2559","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.276085+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:54.276085+0000","last_peered":"2021-01-15T18:41:54.276085+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:54.276085+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2503,"ondisk_log_size":2503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":19999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'2499","reported_seq":"2561","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.023563+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:54.023563+0000","last_peered":"2021-01-15T18:41:54.023563+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:54.023563+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2499,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20431014,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":19959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"34'2510","reported_seq":"2574","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.845632+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:53.845632+0000","last_peered":"2021-01-15T18:41:53.845632+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:53.845632+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2510,"ondisk_log_size":2510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20504742,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1099}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'2679","reported_seq":"2742","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.920305+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:53.920305+0000","last_peered":"2021-01-15T18:41:53.920305+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:53.920305+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2679,"ondisk_log_size":2679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21897382,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'2733","reported_seq":"2791","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.150721+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:54.150721+0000","last_peered":"2021-01-15T18:41:54.150721+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:54.150721+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2733,"ondisk_log_size":2733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22307148,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2542","reported_seq":"2621","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.122121+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:53.122121+0000","last_peered":"2021-01-15T18:41:53.122121+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:53.122121+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2542,"ondisk_log_size":2542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20799654,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":20314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1158}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":336873748,"num_objects":41136,"num_object_clones":0,"num_object_copies":164544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41163,"num_write_kb":329094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4310,"num_bytes_recovered":35028992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":600608768,"data_stored":600608768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41190,"ondisk_log_size":41190,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257256,"kb_used_data":165032,"kb_used_omap":0,"kb_used_meta":92223,"kb_avail":94114584,"statfs":{"total":96636764160,"available":96373334016,"internally_reserved":0,"allocated":168992768,"data_stored":168807063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":94436404},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":0.55800000000000005},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.1739999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246788,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":83456,"kb_avail":94125052,"statfs":{"total":96636764160,"available":96384053248,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":85458944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":0.65000000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243356,"kb_used_data":163876,"kb_used_omap":0,"kb_used_meta":79487,"kb_avail":94128484,"statfs":{"total":96636764160,"available":96387567616,"internally_reserved":0,"allocated":167809024,"data_stored":167623319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81394740},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.54900000000000004},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.63800000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.74299999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155560,"kb_used_data":92576,"kb_used_omap":0,"kb_used_meta":62975,"kb_avail":94216280,"statfs":{"total":96636764160,"available":96477470720,"internally_reserved":0,"allocated":94797824,"data_stored":94612119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":64486452},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.503}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.65600000000000003}]}]}]}} 2021-01-15T18:41:55.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:41:56.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 116 20 41961 41941 2.82383 2.82031 0.0347955 0.0553097 2021-01-15T18:41:56.770 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:56.771 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17097/165100 objects degraded (10.356%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41275,"data_bytes":338012436,"bytes_used":939618304,"bytes_avail":385607438336,"bytes_total":386547056640,"degraded_objects":17097,"degraded_total":165100,"degraded_ratio":0.10355542095699576,"recovering_objects_per_sec":228,"recovering_bytes_per_sec":1870245,"recovering_keys_per_sec":0,"num_objects_recovered":2493,"num_bytes_recovered":20381696,"num_keys_recovered":0,"write_bytes_sec":623916,"read_op_per_sec":0,"write_op_per_sec":76},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:41:56.804 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:41:56.805 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:41:57.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 117 20 42322 42302 2.8238 2.82031 0.0905991 0.0553128 2021-01-15T18:41:57.798 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:41:57.825 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:41:57.883 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2021-01-15T18:41:56.398733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338012436,"num_objects":41275,"num_object_clones":0,"num_object_copies":165100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41302,"num_write_kb":330206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4310,"num_bytes_recovered":35028992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41329,"ondisk_log_size":41329,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":917596,"kb_used_data":589204,"kb_used_omap":2,"kb_used_meta":328381,"kb_avail":376569764,"statfs":{"total":386547056640,"available":385607438336,"internally_reserved":0,"allocated":603344896,"data_stored":602602076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":336262300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6799360,"num_objects":830,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":830,"num_write_kb":6640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":20381696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.897874"},"pg_stats":[{"pgid":"1.f","version":"34'2490","reported_seq":"2534","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.262303+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:54.262303+0000","last_peered":"2021-01-15T18:41:54.262303+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:54.262303+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2490,"ondisk_log_size":2490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20398080,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'2559","reported_seq":"5477","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:54.555710+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:54.555710+0000","last_peered":"2021-01-15T18:41:54.555710+0000","last_clean":"2021-01-15T18:41:54.555710+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:54.555710+0000","last_undegraded":"2021-01-15T18:41:54.555710+0000","last_fullsized":"2021-01-15T18:41:54.555710+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20922534,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2554","reported_seq":"2631","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.191245+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:53.191245+0000","last_peered":"2021-01-15T18:41:53.191245+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:53.191245+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2554,"ondisk_log_size":2554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20893779,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":20410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1135}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"34'2625","reported_seq":"2669","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.222694+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:54.222694+0000","last_peered":"2021-01-15T18:41:54.222694+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:54.222694+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2625,"ondisk_log_size":2625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21504000,"num_objects":2625,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2625,"num_write_kb":21000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"34'2647","reported_seq":"3233","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:54.426615+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:41:54.426615+0000","last_peered":"2021-01-15T18:41:54.426615+0000","last_clean":"2021-01-15T18:41:54.426615+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:41:54.426615+0000","last_undegraded":"2021-01-15T18:41:54.426615+0000","last_fullsized":"2021-01-15T18:41:54.426615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2647,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21635238,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2645,"num_write_kb":21140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"34'2572","reported_seq":"2777","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:54.486909+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:41:54.486909+0000","last_peered":"2021-01-15T18:41:54.486909+0000","last_clean":"2021-01-15T18:41:54.486909+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:54.486909+0000","last_undegraded":"2021-01-15T18:41:54.486909+0000","last_fullsized":"2021-01-15T18:41:54.486909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2572,"ondisk_log_size":2572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21037222,"num_objects":2569,"num_object_clones":0,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"34'2559","reported_seq":"2611","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.200003+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:54.200003+0000","last_peered":"2021-01-15T18:41:54.200003+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:54.200003+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2559,"ondisk_log_size":2559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20922534,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1163}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2588","reported_seq":"2659","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.029390+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:53.029390+0000","last_peered":"2021-01-15T18:41:53.029390+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:53.029390+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2588,"ondisk_log_size":2588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21188691,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":20693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"34'2635","reported_seq":"2675","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.285804+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:54.285804+0000","last_peered":"2021-01-15T18:41:54.285804+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:54.285804+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2635,"ondisk_log_size":2635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21573715,"num_objects":2634,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":21069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"34'2634","reported_seq":"5299","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.300982+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:54.300982+0000","last_peered":"2021-01-15T18:41:54.300982+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:54.300982+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2634,"ondisk_log_size":2634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21524729,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1306,"num_bytes_recovered":10649600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"34'2503","reported_seq":"2559","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.276085+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:54.276085+0000","last_peered":"2021-01-15T18:41:54.276085+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:54.276085+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2503,"ondisk_log_size":2503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20471974,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":19999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'2499","reported_seq":"2561","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.023563+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:54.023563+0000","last_peered":"2021-01-15T18:41:54.023563+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:54.023563+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2499,"ondisk_log_size":2499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20431014,"num_objects":2495,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":19959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"34'2510","reported_seq":"2574","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.845632+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:53.845632+0000","last_peered":"2021-01-15T18:41:53.845632+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:53.845632+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2510,"ondisk_log_size":2510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20504742,"num_objects":2504,"num_object_clones":0,"num_object_copies":10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":20036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1099}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'2679","reported_seq":"2742","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.920305+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:53.920305+0000","last_peered":"2021-01-15T18:41:53.920305+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:53.920305+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2679,"ondisk_log_size":2679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21897382,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'2733","reported_seq":"2791","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:54.150721+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:54.150721+0000","last_peered":"2021-01-15T18:41:54.150721+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:54.150721+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2733,"ondisk_log_size":2733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22307148,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2542","reported_seq":"2621","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:53.122121+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:53.122121+0000","last_peered":"2021-01-15T18:41:53.122121+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:53.122121+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2542,"ondisk_log_size":2542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20799654,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":20314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1158}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":338012436,"num_objects":41275,"num_object_clones":0,"num_object_copies":165100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41302,"num_write_kb":330206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4310,"num_bytes_recovered":35028992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605835264,"data_stored":605835264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41329,"ondisk_log_size":41329,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215133,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257256,"kb_used_data":165032,"kb_used_omap":0,"kb_used_meta":92223,"kb_avail":94114584,"statfs":{"total":96636764160,"available":96373334016,"internally_reserved":0,"allocated":168992768,"data_stored":168807063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":94436404},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":0.55800000000000005},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.1739999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246788,"kb_used_data":163324,"kb_used_omap":0,"kb_used_meta":83456,"kb_avail":94125052,"statfs":{"total":96636764160,"available":96384053248,"internally_reserved":0,"allocated":167243776,"data_stored":167058071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":85458944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":0.65000000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":243356,"kb_used_data":163876,"kb_used_omap":0,"kb_used_meta":79487,"kb_avail":94128484,"statfs":{"total":96636764160,"available":96387567616,"internally_reserved":0,"allocated":167809024,"data_stored":167623319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81394740},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.54900000000000004},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.63800000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:40:52 2021","interfaces":[{"interface":"back","average":{"1min":4.694,"5min":4.694,"15min":4.694},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":41.984999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":13.124000000000001,"5min":13.124000000000001,"15min":13.124000000000001},"last":0.74299999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170196,"kb_used_data":96972,"kb_used_omap":0,"kb_used_meta":73215,"kb_avail":94201644,"statfs":{"total":96636764160,"available":96462483456,"internally_reserved":0,"allocated":99299328,"data_stored":99113623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":74972212},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":16384},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.503}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.65600000000000003}]}]}]}} 2021-01-15T18:41:58.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 118 20 42628 42608 2.82013 2.39062 0.0149234 0.0553385 2021-01-15T18:41:59.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 119 20 42915 42895 2.81527 2.24219 0.0724676 0.055479 2021-01-15T18:42:00.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:42:00.040435+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0555374 2021-01-15T18:42:00.040 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:42:00.041 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 120 20 43232 43212 2.81245 2.47656 0.0409957 0.0555374 2021-01-15T18:42:00.885 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:01.041 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 121 20 43565 43545 2.81069 2.60156 0.0362796 0.0555292 2021-01-15T18:42:01.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:01.561 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:01.584 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2021-01-15T18:42:00.620589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351971604,"num_objects":42979,"num_object_clones":0,"num_object_copies":171916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43006,"num_write_kb":343838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4418,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43033,"ondisk_log_size":43033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":953448,"kb_used_data":616924,"kb_used_omap":2,"kb_used_meta":336573,"kb_avail":376533912,"statfs":{"total":386547056640,"available":385570725888,"internally_reserved":0,"allocated":631730176,"data_stored":630987356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":344650908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,6,10,16,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17948672,"num_objects":2191,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-731,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":17528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":5988352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.226257"},"pg_stats":[{"pgid":"1.f","version":"34'2589","reported_seq":"2632","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.237075+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:59.237075+0000","last_peered":"2021-01-15T18:41:59.237075+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:59.237075+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2589,"ondisk_log_size":2589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21209088,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":20712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'2677","reported_seq":"5595","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.535185+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:59.535185+0000","last_peered":"2021-01-15T18:41:59.535185+0000","last_clean":"2021-01-15T18:41:59.535185+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:59.535185+0000","last_undegraded":"2021-01-15T18:41:59.535185+0000","last_fullsized":"2021-01-15T18:41:59.535185+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2677,"ondisk_log_size":2677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21889190,"num_objects":2673,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2656","reported_seq":"2733","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:57.746069+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:57.746069+0000","last_peered":"2021-01-15T18:41:57.746069+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:57.746069+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2656,"ondisk_log_size":2656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21729363,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"34'2720","reported_seq":"2764","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.186567+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:59.186567+0000","last_peered":"2021-01-15T18:41:59.186567+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:59.186567+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2720,"ondisk_log_size":2720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22282240,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"34'2769","reported_seq":"3355","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.479202+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:41:59.479202+0000","last_peered":"2021-01-15T18:41:59.479202+0000","last_clean":"2021-01-15T18:41:59.479202+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:41:59.479202+0000","last_undegraded":"2021-01-15T18:41:59.479202+0000","last_fullsized":"2021-01-15T18:41:59.479202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2769,"ondisk_log_size":2769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22634662,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"34'2683","reported_seq":"2888","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.535046+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:41:59.535046+0000","last_peered":"2021-01-15T18:41:59.535046+0000","last_clean":"2021-01-15T18:41:59.535046+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:59.535046+0000","last_undegraded":"2021-01-15T18:41:59.535046+0000","last_fullsized":"2021-01-15T18:41:59.535046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2683,"ondisk_log_size":2683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21946534,"num_objects":2680,"num_object_clones":0,"num_object_copies":10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":21439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"34'2657","reported_seq":"2709","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.198629+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:59.198629+0000","last_peered":"2021-01-15T18:41:59.198629+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:59.198629+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2657,"ondisk_log_size":2657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21725350,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2694","reported_seq":"2765","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.180474+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:58.180474+0000","last_peered":"2021-01-15T18:41:58.180474+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:58.180474+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2694,"ondisk_log_size":2694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22057043,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"34'2737","reported_seq":"2777","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.185865+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:59.185865+0000","last_peered":"2021-01-15T18:41:59.185865+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:59.185865+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2737,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22409299,"num_objects":2736,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":21885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1300}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"34'2747","reported_seq":"5628","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.316860+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:59.316860+0000","last_peered":"2021-01-15T18:41:59.316860+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:59.316860+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2747,"ondisk_log_size":2747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22450425,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":21932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1414,"num_bytes_recovered":11534336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"34'2626","reported_seq":"2682","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.309997+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:59.309997+0000","last_peered":"2021-01-15T18:41:59.309997+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:59.309997+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2626,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21479590,"num_objects":2623,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'2607","reported_seq":"2669","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.983186+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:58.983186+0000","last_peered":"2021-01-15T18:41:58.983186+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:58.983186+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2607,"ondisk_log_size":2607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21315750,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"34'2606","reported_seq":"2670","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.976001+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:58.976001+0000","last_peered":"2021-01-15T18:41:58.976001+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:58.976001+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2606,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21291174,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'2788","reported_seq":"2851","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.920563+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:58.920563+0000","last_peered":"2021-01-15T18:41:58.920563+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:58.920563+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2788,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22790310,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'2830","reported_seq":"2888","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.305167+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:59.305167+0000","last_peered":"2021-01-15T18:41:59.305167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:59.305167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2830,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23101772,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2647","reported_seq":"2726","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.178044+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:58.178044+0000","last_peered":"2021-01-15T18:41:58.178044+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:58.178044+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2647,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21659814,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2645,"num_write_kb":21154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":351971604,"num_objects":42979,"num_object_clones":0,"num_object_copies":171916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43006,"num_write_kb":343838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4418,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":633950208,"data_stored":633950208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43033,"ondisk_log_size":43033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254944,"kb_used_data":171928,"kb_used_omap":0,"kb_used_meta":83007,"kb_avail":94116896,"statfs":{"total":96636764160,"available":96375701504,"internally_reserved":0,"allocated":176054272,"data_stored":175868567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84999220},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":7.1289999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":7.234}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.252}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.081},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":2.0030000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264680,"kb_used_data":170100,"kb_used_omap":0,"kb_used_meta":94656,"kb_avail":94107160,"statfs":{"total":96636764160,"available":96365731840,"internally_reserved":0,"allocated":174182400,"data_stored":173996695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96927744},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.4450000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":2.8900000000000001},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":2.9860000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":7.1470000000000002},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":2.0830000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280544,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":261580,"kb_used_data":170820,"kb_used_omap":0,"kb_used_meta":90751,"kb_avail":94110260,"statfs":{"total":96636764160,"available":96368906240,"internally_reserved":0,"allocated":174919680,"data_stored":174733975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92929076},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.98},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.433}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":1.732},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.5090000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":9.9559999999999995},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":10.054}]}]},{"osd":1,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172244,"kb_used_data":104076,"kb_used_omap":0,"kb_used_meta":68159,"kb_avail":94199596,"statfs":{"total":96636764160,"available":96460386304,"internally_reserved":0,"allocated":106573824,"data_stored":106388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":69794868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":2.448},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":2.0329999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.3640000000000001}]}]}]}} 2021-01-15T18:42:01.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:02.041 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 122 20 43804 43784 2.80295 1.86719 0.0448861 0.0557331 2021-01-15T18:42:02.700 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:02.701 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16989/171916 objects degraded (9.882%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":173,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42979,"data_bytes":351971604,"bytes_used":976330752,"bytes_avail":385570725888,"bytes_total":386547056640,"degraded_objects":16989,"degraded_total":171916,"degraded_ratio":0.098821517485283505,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":110561,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"write_bytes_sec":1886715,"read_op_per_sec":0,"write_op_per_sec":230},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:02.752 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:02.752 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:03.042 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 123 20 44141 44121 2.80153 2.63281 0.0416235 0.0557587 2021-01-15T18:42:03.711 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:03.734 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:03.786 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2021-01-15T18:42:02.400330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351971604,"num_objects":42979,"num_object_clones":0,"num_object_copies":171916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43006,"num_write_kb":343838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4418,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43033,"ondisk_log_size":43033,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":953448,"kb_used_data":616924,"kb_used_omap":2,"kb_used_meta":336573,"kb_avail":376533912,"statfs":{"total":386547056640,"available":385570725888,"internally_reserved":0,"allocated":631730176,"data_stored":630987356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":344650908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,6,10,16,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15097856,"num_objects":1843,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1843,"num_write_kb":14744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002190"},"pg_stats":[{"pgid":"1.f","version":"34'2589","reported_seq":"2632","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.237075+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:41:59.237075+0000","last_peered":"2021-01-15T18:41:59.237075+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:41:59.237075+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2589,"ondisk_log_size":2589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21209088,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":20712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"34'2677","reported_seq":"5595","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.535185+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:41:59.535185+0000","last_peered":"2021-01-15T18:41:59.535185+0000","last_clean":"2021-01-15T18:41:59.535185+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:41:59.535185+0000","last_undegraded":"2021-01-15T18:41:59.535185+0000","last_fullsized":"2021-01-15T18:41:59.535185+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2677,"ondisk_log_size":2677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21889190,"num_objects":2673,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":21388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"34'2656","reported_seq":"2733","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:57.746069+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:41:57.746069+0000","last_peered":"2021-01-15T18:41:57.746069+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:41:57.746069+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2656,"ondisk_log_size":2656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21729363,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"34'2720","reported_seq":"2764","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.186567+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:41:59.186567+0000","last_peered":"2021-01-15T18:41:59.186567+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:41:59.186567+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2720,"ondisk_log_size":2720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22282240,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1288}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"34'2769","reported_seq":"3355","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.479202+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:41:59.479202+0000","last_peered":"2021-01-15T18:41:59.479202+0000","last_clean":"2021-01-15T18:41:59.479202+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:41:59.479202+0000","last_undegraded":"2021-01-15T18:41:59.479202+0000","last_fullsized":"2021-01-15T18:41:59.479202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2769,"ondisk_log_size":2769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22634662,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"34'2683","reported_seq":"2888","reported_epoch":"34","state":"active+clean","last_fresh":"2021-01-15T18:41:59.535046+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:41:59.535046+0000","last_peered":"2021-01-15T18:41:59.535046+0000","last_clean":"2021-01-15T18:41:59.535046+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:41:59.535046+0000","last_undegraded":"2021-01-15T18:41:59.535046+0000","last_fullsized":"2021-01-15T18:41:59.535046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2683,"ondisk_log_size":2683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21946534,"num_objects":2680,"num_object_clones":0,"num_object_copies":10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":21439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"34'2657","reported_seq":"2709","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.198629+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:41:59.198629+0000","last_peered":"2021-01-15T18:41:59.198629+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:41:59.198629+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2657,"ondisk_log_size":2657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21725350,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"34'2694","reported_seq":"2765","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.180474+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:41:58.180474+0000","last_peered":"2021-01-15T18:41:58.180474+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:41:58.180474+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2694,"ondisk_log_size":2694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22057043,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2693,"num_write_kb":21541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"34'2737","reported_seq":"2777","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.185865+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:41:59.185865+0000","last_peered":"2021-01-15T18:41:59.185865+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:41:59.185865+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2737,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22409299,"num_objects":2736,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":21885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1300}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"34'2747","reported_seq":"5628","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.316860+0000","last_change":"2021-01-15T18:41:43.938301+0000","last_active":"2021-01-15T18:41:59.316860+0000","last_peered":"2021-01-15T18:41:59.316860+0000","last_clean":"2021-01-15T18:39:34.840441+0000","last_became_active":"2021-01-15T18:40:32.729880+0000","last_became_peered":"2021-01-15T18:40:32.729880+0000","last_unstale":"2021-01-15T18:41:59.316860+0000","last_undegraded":"2021-01-15T18:40:32.530154+0000","last_fullsized":"2021-01-15T18:40:32.483199+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2747,"ondisk_log_size":2747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22450425,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":21932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1414,"num_bytes_recovered":11534336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"34'2626","reported_seq":"2682","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.309997+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:41:59.309997+0000","last_peered":"2021-01-15T18:41:59.309997+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:41:59.309997+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2626,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21479590,"num_objects":2623,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'2607","reported_seq":"2669","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.983186+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:41:58.983186+0000","last_peered":"2021-01-15T18:41:58.983186+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:41:58.983186+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2607,"ondisk_log_size":2607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21315750,"num_objects":2603,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"34'2606","reported_seq":"2670","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.976001+0000","last_change":"2021-01-15T18:40:32.757315+0000","last_active":"2021-01-15T18:41:58.976001+0000","last_peered":"2021-01-15T18:41:58.976001+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:41:58.976001+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2606,"ondisk_log_size":2606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21291174,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'2788","reported_seq":"2851","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.920563+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:41:58.920563+0000","last_peered":"2021-01-15T18:41:58.920563+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:41:58.920563+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2788,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22790310,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1341}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"34'2830","reported_seq":"2888","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:59.305167+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:41:59.305167+0000","last_peered":"2021-01-15T18:41:59.305167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:41:59.305167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2830,"ondisk_log_size":2830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23101772,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1336}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"34'2647","reported_seq":"2726","reported_epoch":"34","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:41:58.178044+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:41:58.178044+0000","last_peered":"2021-01-15T18:41:58.178044+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:41:58.178044+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2647,"ondisk_log_size":2647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21659814,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2645,"num_write_kb":21154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":351971604,"num_objects":42979,"num_object_clones":0,"num_object_copies":171916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43006,"num_write_kb":343838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4418,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":633950208,"data_stored":633950208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43033,"ondisk_log_size":43033,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215134,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254944,"kb_used_data":171928,"kb_used_omap":0,"kb_used_meta":83007,"kb_avail":94116896,"statfs":{"total":96636764160,"available":96375701504,"internally_reserved":0,"allocated":176054272,"data_stored":175868567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84999220},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":7.1289999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":7.234}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.252}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.081},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":2.0030000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":264680,"kb_used_data":170100,"kb_used_omap":0,"kb_used_meta":94656,"kb_avail":94107160,"statfs":{"total":96636764160,"available":96365731840,"internally_reserved":0,"allocated":174182400,"data_stored":173996695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96927744},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.4450000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":2.8900000000000001},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":2.9860000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":7.1470000000000002},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":2.0830000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280544,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":261580,"kb_used_data":170820,"kb_used_omap":0,"kb_used_meta":90751,"kb_avail":94110260,"statfs":{"total":96636764160,"available":96368906240,"internally_reserved":0,"allocated":174919680,"data_stored":174733975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92929076},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.98},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.433}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":1.732},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.5090000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":9.9559999999999995},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":10.054}]}]},{"osd":1,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172244,"kb_used_data":104076,"kb_used_omap":0,"kb_used_meta":68159,"kb_avail":94199596,"statfs":{"total":96636764160,"available":96460386304,"internally_reserved":0,"allocated":106573824,"data_stored":106388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":69794868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":2.448},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":2.0329999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.3640000000000001}]}]}]}} 2021-01-15T18:42:04.042 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 124 20 44462 44442 2.79916 2.50781 0.0941401 0.0557964 2021-01-15T18:42:05.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 125 20 44817 44797 2.79895 2.77344 0.0396404 0.0558051 2021-01-15T18:42:06.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 126 20 45174 45154 2.79887 2.78906 0.0420367 0.0558091 2021-01-15T18:42:06.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:07.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 127 20 45520 45500 2.79812 2.70312 0.0912946 0.0558275 2021-01-15T18:42:07.930 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:07.949 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:07.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2021-01-15T18:42:06.401601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364538132,"num_objects":44513,"num_object_clones":0,"num_object_copies":178052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44540,"num_write_kb":356110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4509,"num_bytes_recovered":36659200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44567,"ondisk_log_size":44567,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":989704,"kb_used_data":641708,"kb_used_omap":2,"kb_used_meta":347965,"kb_avail":376497656,"statfs":{"total":386547056640,"available":385533599744,"internally_reserved":0,"allocated":657108992,"data_stored":656328504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":356316316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,15,6,13,13,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23961600,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2925,"num_write_kb":23400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002331"},"pg_stats":[{"pgid":"1.f","version":"36'2674","reported_seq":"2720","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.164913+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:04.164913+0000","last_peered":"2021-01-15T18:42:04.164913+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:04.164913+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2674,"ondisk_log_size":2674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21905408,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2674,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'2767","reported_seq":"5687","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.393027+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:04.393027+0000","last_peered":"2021-01-15T18:42:04.393027+0000","last_clean":"2021-01-15T18:42:04.393027+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:04.393027+0000","last_undegraded":"2021-01-15T18:42:04.393027+0000","last_fullsized":"2021-01-15T18:42:04.393027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2767,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22626470,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'2742","reported_seq":"2820","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:03.129460+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:03.129460+0000","last_peered":"2021-01-15T18:42:03.129460+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:03.129460+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2742,"ondisk_log_size":2742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22433875,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2741,"num_write_kb":21914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1323}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'2826","reported_seq":"2872","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.273376+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:04.273376+0000","last_peered":"2021-01-15T18:42:04.273376+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:04.273376+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2826,"ondisk_log_size":2826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23150592,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":22608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'2863","reported_seq":"3451","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.543778+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:04.543778+0000","last_peered":"2021-01-15T18:42:04.543778+0000","last_clean":"2021-01-15T18:42:04.543778+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:04.543778+0000","last_undegraded":"2021-01-15T18:42:04.543778+0000","last_fullsized":"2021-01-15T18:42:04.543778+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2863,"ondisk_log_size":2863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23404710,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2861,"num_write_kb":22868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'2773","reported_seq":"2980","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.507345+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:04.507345+0000","last_peered":"2021-01-15T18:42:04.507345+0000","last_clean":"2021-01-15T18:42:04.507345+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:04.507345+0000","last_undegraded":"2021-01-15T18:42:04.507345+0000","last_fullsized":"2021-01-15T18:42:04.507345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2773,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22683814,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":22159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'2761","reported_seq":"2815","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.251939+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:04.251939+0000","last_peered":"2021-01-15T18:42:04.251939+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:04.251939+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2761,"ondisk_log_size":2761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22577318,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":22060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1365}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'2787","reported_seq":"2861","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:03.147862+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:03.147862+0000","last_peered":"2021-01-15T18:42:03.147862+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:03.147862+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2787,"ondisk_log_size":2787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22818899,"num_objects":2786,"num_object_clones":0,"num_object_copies":11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1334}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'2849","reported_seq":"2890","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.251984+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:04.251984+0000","last_peered":"2021-01-15T18:42:04.251984+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:04.251984+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2849,"ondisk_log_size":2849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23326803,"num_objects":2848,"num_object_clones":0,"num_object_copies":11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'2841","reported_seq":"5778","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.294334+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:04.294334+0000","last_peered":"2021-01-15T18:42:04.294334+0000","last_clean":"2021-01-15T18:42:04.294334+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:04.294334+0000","last_undegraded":"2021-01-15T18:42:04.294334+0000","last_fullsized":"2021-01-15T18:42:04.294334+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2841,"ondisk_log_size":2841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23220473,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":22684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'2722","reported_seq":"2780","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.276083+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:04.276083+0000","last_peered":"2021-01-15T18:42:04.276083+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:04.276083+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2722,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22266022,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'2701","reported_seq":"2764","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.010715+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:04.010715+0000","last_peered":"2021-01-15T18:42:04.010715+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:04.010715+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2701,"ondisk_log_size":2701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22085798,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":21575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'2703","reported_seq":"2908","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.050099+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:04.050099+0000","last_peered":"2021-01-15T18:42:04.050099+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:04.050099+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2703,"ondisk_log_size":2703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22085798,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2701,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":565248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'2883","reported_seq":"2948","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.015135+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:04.015135+0000","last_peered":"2021-01-15T18:42:04.015135+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:04.015135+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2883,"ondisk_log_size":2883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23568550,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2881,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'2938","reported_seq":"2998","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.303997+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:04.303997+0000","last_peered":"2021-01-15T18:42:04.303997+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:04.303997+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2938,"ondisk_log_size":2938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23986508,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":23443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'2737","reported_seq":"2818","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:03.196917+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:03.196917+0000","last_peered":"2021-01-15T18:42:03.196917+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:03.196917+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2737,"ondisk_log_size":2737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22397094,"num_objects":2735,"num_object_clones":0,"num_object_copies":10940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":21874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":364538132,"num_objects":44513,"num_object_clones":0,"num_object_copies":178052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44540,"num_write_kb":356110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4509,"num_bytes_recovered":36659200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":659628032,"data_stored":659628032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44567,"ondisk_log_size":44567,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272108,"kb_used_data":178084,"kb_used_omap":0,"kb_used_meta":94015,"kb_avail":94099732,"statfs":{"total":96636764160,"available":96358125568,"internally_reserved":0,"allocated":182358016,"data_stored":182162894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":96271412},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.5299999999999998},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.629}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.252}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.7370000000000001},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.89900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265140,"kb_used_data":175984,"kb_used_omap":0,"kb_used_meta":89152,"kb_avail":94106700,"statfs":{"total":96636764160,"available":96365260800,"internally_reserved":0,"allocated":180207616,"data_stored":180012494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91291648},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":16.326000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.819}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":13.926},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":2.1600000000000001},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":6.2720000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262344,"kb_used_data":176960,"kb_used_omap":0,"kb_used_meta":85375,"kb_avail":94109496,"statfs":{"total":96636764160,"available":96368123904,"internally_reserved":0,"allocated":181207040,"data_stored":181011918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87424052},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":7.085},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":2.7639999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":6.9779999999999998},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.3360000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.911},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":1.9990000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":190112,"kb_used_data":110680,"kb_used_omap":0,"kb_used_meta":79423,"kb_avail":94181728,"statfs":{"total":96636764160,"available":96442089472,"internally_reserved":0,"allocated":113336320,"data_stored":113141198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81329204},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":9.3499999999999996},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.502}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":9.5519999999999996},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.119},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.415}]}]}]}} 2021-01-15T18:42:07.994 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:08.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 128 20 45861 45841 2.79707 2.66406 0.0640784 0.0558495 2021-01-15T18:42:09.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 129 20 46181 46161 2.79476 2.5 0.0423778 0.05589 2021-01-15T18:42:09.343 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:09.344 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16896/179276 objects degraded (9.425%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":179,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44819,"data_bytes":367044884,"bytes_used":1032065024,"bytes_avail":385514991616,"bytes_total":386547056640,"degraded_objects":16896,"degraded_total":179276,"degraded_ratio":0.094245743992503184,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":93157,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":745472,"num_keys_recovered":0,"write_bytes_sec":1883615,"read_op_per_sec":0,"write_op_per_sec":229},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:09.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:10.043 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 130 20 46493 46473 2.79201 2.4375 0.0446695 0.0559449 2021-01-15T18:42:10.143 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:10.171 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2021-01-15T18:42:08.402119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367044884,"num_objects":44819,"num_object_clones":0,"num_object_copies":179276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44846,"num_write_kb":358558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4509,"num_bytes_recovered":36659200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44873,"ondisk_log_size":44873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1007876,"kb_used_data":648612,"kb_used_omap":2,"kb_used_meta":359229,"kb_avail":376479484,"statfs":{"total":386547056640,"available":385514991616,"internally_reserved":0,"allocated":664178688,"data_stored":663398200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":367850652},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,11,4,14,13,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15073280,"num_objects":1840,"num_object_clones":0,"num_object_copies":7360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1840,"num_write_kb":14720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002312"},"pg_stats":[{"pgid":"1.f","version":"36'2674","reported_seq":"2720","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.164913+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:04.164913+0000","last_peered":"2021-01-15T18:42:04.164913+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:04.164913+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2674,"ondisk_log_size":2674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21905408,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2674,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'2767","reported_seq":"5687","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.393027+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:04.393027+0000","last_peered":"2021-01-15T18:42:04.393027+0000","last_clean":"2021-01-15T18:42:04.393027+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:04.393027+0000","last_undegraded":"2021-01-15T18:42:04.393027+0000","last_fullsized":"2021-01-15T18:42:04.393027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2767,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22626470,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'2850","reported_seq":"2928","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.156975+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:08.156975+0000","last_peered":"2021-01-15T18:42:08.156975+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:08.156975+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2850,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23318611,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2849,"num_write_kb":22778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'2826","reported_seq":"2872","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.273376+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:04.273376+0000","last_peered":"2021-01-15T18:42:04.273376+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:04.273376+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2826,"ondisk_log_size":2826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23150592,"num_objects":2826,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":22608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'2863","reported_seq":"3451","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.543778+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:04.543778+0000","last_peered":"2021-01-15T18:42:04.543778+0000","last_clean":"2021-01-15T18:42:04.543778+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:04.543778+0000","last_undegraded":"2021-01-15T18:42:04.543778+0000","last_fullsized":"2021-01-15T18:42:04.543778+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2863,"ondisk_log_size":2863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23404710,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2861,"num_write_kb":22868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'2773","reported_seq":"2980","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.507345+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:04.507345+0000","last_peered":"2021-01-15T18:42:04.507345+0000","last_clean":"2021-01-15T18:42:04.507345+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:04.507345+0000","last_undegraded":"2021-01-15T18:42:04.507345+0000","last_fullsized":"2021-01-15T18:42:04.507345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2773,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22683814,"num_objects":2770,"num_object_clones":0,"num_object_copies":11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":22159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'2761","reported_seq":"2815","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.251939+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:04.251939+0000","last_peered":"2021-01-15T18:42:04.251939+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:04.251939+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2761,"ondisk_log_size":2761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22577318,"num_objects":2757,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":22060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1365}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'2880","reported_seq":"2954","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.124456+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:08.124456+0000","last_peered":"2021-01-15T18:42:08.124456+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:08.124456+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2880,"ondisk_log_size":2880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23580755,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'2849","reported_seq":"2890","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.251984+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:04.251984+0000","last_peered":"2021-01-15T18:42:04.251984+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:04.251984+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2849,"ondisk_log_size":2849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23326803,"num_objects":2848,"num_object_clones":0,"num_object_copies":11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":22781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'2841","reported_seq":"5778","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:04.294334+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:04.294334+0000","last_peered":"2021-01-15T18:42:04.294334+0000","last_clean":"2021-01-15T18:42:04.294334+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:04.294334+0000","last_undegraded":"2021-01-15T18:42:04.294334+0000","last_fullsized":"2021-01-15T18:42:04.294334+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2841,"ondisk_log_size":2841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23220473,"num_objects":2836,"num_object_clones":0,"num_object_copies":11344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":22684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'2722","reported_seq":"2780","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.276083+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:04.276083+0000","last_peered":"2021-01-15T18:42:04.276083+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:04.276083+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2722,"ondisk_log_size":2722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22266022,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":21751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'2701","reported_seq":"2764","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.010715+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:04.010715+0000","last_peered":"2021-01-15T18:42:04.010715+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:04.010715+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2701,"ondisk_log_size":2701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22085798,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":21575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'2703","reported_seq":"2908","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.050099+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:04.050099+0000","last_peered":"2021-01-15T18:42:04.050099+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:04.050099+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2703,"ondisk_log_size":2703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22085798,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2701,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":565248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'2883","reported_seq":"2948","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.015135+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:04.015135+0000","last_peered":"2021-01-15T18:42:04.015135+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:04.015135+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2883,"ondisk_log_size":2883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23568550,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2881,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'2938","reported_seq":"2998","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:04.303997+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:04.303997+0000","last_peered":"2021-01-15T18:42:04.303997+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:04.303997+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2938,"ondisk_log_size":2938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23986508,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":23443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'2842","reported_seq":"2922","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.181879+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:08.181879+0000","last_peered":"2021-01-15T18:42:08.181879+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:08.181879+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2842,"ondisk_log_size":2842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23257254,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":22714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367044884,"num_objects":44819,"num_object_clones":0,"num_object_copies":179276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44846,"num_write_kb":358558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4509,"num_bytes_recovered":36659200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":666644480,"data_stored":666644480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44873,"ondisk_log_size":44873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272108,"kb_used_data":178084,"kb_used_omap":0,"kb_used_meta":94015,"kb_avail":94099732,"statfs":{"total":96636764160,"available":96358125568,"internally_reserved":0,"allocated":182358016,"data_stored":182162894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":96271412},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,4,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.5299999999999998},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.629}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":1.252}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.7370000000000001},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.89900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283312,"kb_used_data":182888,"kb_used_omap":0,"kb_used_meta":100416,"kb_avail":94088528,"statfs":{"total":96636764160,"available":96346652672,"internally_reserved":0,"allocated":187277312,"data_stored":187082190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":102825984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":9.1920000000000002},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":2.9609999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":10.584},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":10.685}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":9.8079999999999998},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":9.548}]}]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262344,"kb_used_data":176960,"kb_used_omap":0,"kb_used_meta":85375,"kb_avail":94109496,"statfs":{"total":96636764160,"available":96368123904,"internally_reserved":0,"allocated":181207040,"data_stored":181011918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87424052},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":7.085},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":2.7639999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":6.9779999999999998},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.3360000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.911},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":1.9990000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":190112,"kb_used_data":110680,"kb_used_omap":0,"kb_used_meta":79423,"kb_avail":94181728,"statfs":{"total":96636764160,"available":96442089472,"internally_reserved":0,"allocated":113336320,"data_stored":113141198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":81329204},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":9.3499999999999996},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.502}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":9.5519999999999996},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.119},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.415}]}]}]}} 2021-01-15T18:42:11.044 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 131 20 46841 46821 2.79144 2.71875 0.0485597 0.0559598 2021-01-15T18:42:12.044 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 132 20 47135 47115 2.78768 2.29688 0.0539096 0.0560334 2021-01-15T18:42:13.044 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 133 20 47452 47432 2.78534 2.47656 0.152829 0.0560723 2021-01-15T18:42:13.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:14.045 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 134 20 47769 47749 2.78304 2.47656 0.0290512 0.0561161 2021-01-15T18:42:14.116 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:14.118 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:14.129 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2021-01-15T18:42:12.404820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":378636564,"num_objects":46234,"num_object_clones":0,"num_object_copies":184936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46261,"num_write_kb":369878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4591,"num_bytes_recovered":37330944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46288,"ondisk_log_size":46288,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1042436,"kb_used_data":669680,"kb_used_omap":2,"kb_used_meta":372733,"kb_avail":376444924,"statfs":{"total":386547056640,"available":385479602176,"internally_reserved":0,"allocated":685752320,"data_stored":684971832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":381678748},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,10,21,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26664960,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":26040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.777951"},"pg_stats":[{"pgid":"1.f","version":"36'2769","reported_seq":"2815","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.261342+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:09.261342+0000","last_peered":"2021-01-15T18:42:09.261342+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:09.261342+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2769,"ondisk_log_size":2769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22683648,"num_objects":2769,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'2882","reported_seq":"5802","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.572908+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:09.572908+0000","last_peered":"2021-01-15T18:42:09.572908+0000","last_clean":"2021-01-15T18:42:09.572908+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:09.572908+0000","last_undegraded":"2021-01-15T18:42:09.572908+0000","last_fullsized":"2021-01-15T18:42:09.572908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2882,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23568550,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'2850","reported_seq":"2928","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.156975+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:08.156975+0000","last_peered":"2021-01-15T18:42:08.156975+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:08.156975+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2850,"ondisk_log_size":2850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23318611,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2849,"num_write_kb":22778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'2930","reported_seq":"2976","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.282504+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:09.282504+0000","last_peered":"2021-01-15T18:42:09.282504+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:09.282504+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2930,"ondisk_log_size":2930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24002560,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2930,"num_write_kb":23440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'2966","reported_seq":"3554","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.503574+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:09.503574+0000","last_peered":"2021-01-15T18:42:09.503574+0000","last_clean":"2021-01-15T18:42:09.503574+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:09.503574+0000","last_undegraded":"2021-01-15T18:42:09.503574+0000","last_fullsized":"2021-01-15T18:42:09.503574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2966,"ondisk_log_size":2966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248486,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":23692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'2871","reported_seq":"3076","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.493124+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:09.493124+0000","last_peered":"2021-01-15T18:42:09.493124+0000","last_clean":"2021-01-15T18:42:09.493124+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:09.493124+0000","last_undegraded":"2021-01-15T18:42:09.493124+0000","last_fullsized":"2021-01-15T18:42:09.493124+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2871,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23486630,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'2866","reported_seq":"2920","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.298388+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:09.298388+0000","last_peered":"2021-01-15T18:42:09.298388+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:09.298388+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2866,"ondisk_log_size":2866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23437478,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1470}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'2880","reported_seq":"2954","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.124456+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:08.124456+0000","last_peered":"2021-01-15T18:42:08.124456+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:08.124456+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2880,"ondisk_log_size":2880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23580755,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'2974","reported_seq":"3015","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.325331+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:09.325331+0000","last_peered":"2021-01-15T18:42:09.325331+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:09.325331+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2974,"ondisk_log_size":2974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24350803,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'2946","reported_seq":"5883","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.077553+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:09.077553+0000","last_peered":"2021-01-15T18:42:09.077553+0000","last_clean":"2021-01-15T18:42:09.077553+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:09.077553+0000","last_undegraded":"2021-01-15T18:42:09.077553+0000","last_fullsized":"2021-01-15T18:42:09.077553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2946,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24080633,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2943,"num_write_kb":23524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'2834","reported_seq":"2890","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.312600+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:09.312600+0000","last_peered":"2021-01-15T18:42:09.312600+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:09.312600+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23183526,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":22647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'2822","reported_seq":"2886","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.064594+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:09.064594+0000","last_peered":"2021-01-15T18:42:09.064594+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:09.064594+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2822,"ondisk_log_size":2822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23077030,"num_objects":2818,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2820,"num_write_kb":22543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'2814","reported_seq":"3184","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.069059+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:09.069059+0000","last_peered":"2021-01-15T18:42:09.069059+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:09.069059+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2814,"ondisk_log_size":2814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22995110,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":22468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1554}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3003","reported_seq":"3068","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.031014+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:09.031014+0000","last_peered":"2021-01-15T18:42:09.031014+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:09.031014+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3003,"ondisk_log_size":3003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24551590,"num_objects":2998,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3001,"num_write_kb":23988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3039","reported_seq":"3099","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:09.268160+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:09.268160+0000","last_peered":"2021-01-15T18:42:09.268160+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:09.268160+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3039,"ondisk_log_size":3039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24813900,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3036,"num_write_kb":24251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'2842","reported_seq":"2922","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:08.181879+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:08.181879+0000","last_peered":"2021-01-15T18:42:08.181879+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:08.181879+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2842,"ondisk_log_size":2842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23257254,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":22714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":378636564,"num_objects":46234,"num_object_clones":0,"num_object_copies":184936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46261,"num_write_kb":369878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4591,"num_bytes_recovered":37330944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":688082944,"data_stored":688082944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46288,"ondisk_log_size":46288,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215136,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":290492,"kb_used_data":185144,"kb_used_omap":0,"kb_used_meta":105343,"kb_avail":94081348,"statfs":{"total":96636764160,"available":96339300352,"internally_reserved":0,"allocated":189587456,"data_stored":189392334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":107871284},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":2.6680000000000001},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":2.7669999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":7.8650000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":3.7429999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":7.4130000000000003},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":2.5129999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":283312,"kb_used_data":182888,"kb_used_omap":0,"kb_used_meta":100416,"kb_avail":94088528,"statfs":{"total":96636764160,"available":96346652672,"internally_reserved":0,"allocated":187277312,"data_stored":187082190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":102825984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":9.1920000000000002},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":2.9609999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":10.584},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":10.685}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":9.8079999999999998},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":9.548}]}]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280460,"kb_used_data":183820,"kb_used_omap":0,"kb_used_meta":96639,"kb_avail":94091380,"statfs":{"total":96636764160,"available":96349573120,"internally_reserved":0,"allocated":188231680,"data_stored":188036558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":98958388},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":0.69599999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.57699999999999996},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":0.874}]}]},{"osd":1,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188172,"kb_used_data":117828,"kb_used_omap":0,"kb_used_meta":70335,"kb_avail":94183668,"statfs":{"total":96636764160,"available":96444076032,"internally_reserved":0,"allocated":120655872,"data_stored":120460750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":72023092},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":1.252},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":6.6130000000000004},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":6.915}]}]}]}} 2021-01-15T18:42:14.131 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:15.045 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 135 20 48051 48031 2.77874 2.20312 0.0394934 0.0562085 2021-01-15T18:42:15.156 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:15.156 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16712/190072 objects degraded (8.792%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":185,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47518,"data_bytes":389155092,"bytes_used":1076518912,"bytes_avail":385470537728,"bytes_total":386547056640,"degraded_objects":16712,"degraded_total":190072,"degraded_ratio":0.087924575950166256,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":187655,"recovering_keys_per_sec":0,"num_objects_recovered":275,"num_bytes_recovered":2252800,"num_keys_recovered":0,"write_bytes_sec":3097345,"read_op_per_sec":0,"write_op_per_sec":378},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:15.233 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:15.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:16.045 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 136 20 48383 48363 2.77738 2.59375 0.0549617 0.0562424 2021-01-15T18:42:16.387 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:16.394 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2021-01-15T18:42:14.405283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389155092,"num_objects":47518,"num_object_clones":0,"num_object_copies":190072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47545,"num_write_kb":380150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4693,"num_bytes_recovered":38166528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47572,"ondisk_log_size":47572,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1051288,"kb_used_data":689976,"kb_used_omap":2,"kb_used_meta":361277,"kb_avail":376436072,"statfs":{"total":386547056640,"available":385470537728,"internally_reserved":0,"allocated":706535424,"data_stored":705754936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":369947804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,6,12,12,4,1,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37183488,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":36312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":2252800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004952"},"pg_stats":[{"pgid":"1.f","version":"36'2868","reported_seq":"2914","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.257987+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:14.257987+0000","last_peered":"2021-01-15T18:42:14.257987+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:14.257987+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2868,"ondisk_log_size":2868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23494656,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":22944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'2882","reported_seq":"5802","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.572908+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:09.572908+0000","last_peered":"2021-01-15T18:42:09.572908+0000","last_clean":"2021-01-15T18:42:09.572908+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:09.572908+0000","last_undegraded":"2021-01-15T18:42:09.572908+0000","last_fullsized":"2021-01-15T18:42:09.572908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2882,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23568550,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'2946","reported_seq":"3024","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:13.154563+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:13.154563+0000","last_peered":"2021-01-15T18:42:13.154563+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:13.154563+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2946,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24105043,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1527}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3035","reported_seq":"3079","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.316184+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:14.316184+0000","last_peered":"2021-01-15T18:42:14.316184+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:14.316184+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3035,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24862720,"num_objects":3035,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3035,"num_write_kb":24280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'2966","reported_seq":"3554","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.503574+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:09.503574+0000","last_peered":"2021-01-15T18:42:09.503574+0000","last_clean":"2021-01-15T18:42:09.503574+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:09.503574+0000","last_undegraded":"2021-01-15T18:42:09.503574+0000","last_fullsized":"2021-01-15T18:42:09.503574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2966,"ondisk_log_size":2966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248486,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":23692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'2871","reported_seq":"3076","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:09.493124+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:09.493124+0000","last_peered":"2021-01-15T18:42:09.493124+0000","last_clean":"2021-01-15T18:42:09.493124+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:09.493124+0000","last_undegraded":"2021-01-15T18:42:09.493124+0000","last_fullsized":"2021-01-15T18:42:09.493124+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2871,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23486630,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'2955","reported_seq":"3007","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.243002+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:14.243002+0000","last_peered":"2021-01-15T18:42:14.243002+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:14.243002+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2955,"ondisk_log_size":2955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24166566,"num_objects":2951,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'2990","reported_seq":"3063","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:13.208768+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:13.208768+0000","last_peered":"2021-01-15T18:42:13.208768+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:13.208768+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2990,"ondisk_log_size":2990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24481875,"num_objects":2989,"num_object_clones":0,"num_object_copies":11956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":23909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1537}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3067","reported_seq":"3109","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:13.963023+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:13.963023+0000","last_peered":"2021-01-15T18:42:13.963023+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:13.963023+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3067,"ondisk_log_size":3067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25112659,"num_objects":3066,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3066,"num_write_kb":24525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3031","reported_seq":"5968","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:14.269024+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:14.269024+0000","last_peered":"2021-01-15T18:42:14.269024+0000","last_clean":"2021-01-15T18:42:14.269024+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:14.269024+0000","last_undegraded":"2021-01-15T18:42:14.269024+0000","last_fullsized":"2021-01-15T18:42:14.269024+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24776953,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3028,"num_write_kb":24204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'2937","reported_seq":"2995","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.278236+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:14.278236+0000","last_peered":"2021-01-15T18:42:14.278236+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:14.278236+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2937,"ondisk_log_size":2937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24027302,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":23471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'2937","reported_seq":"3001","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.042602+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:14.042602+0000","last_peered":"2021-01-15T18:42:14.042602+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:14.042602+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2937,"ondisk_log_size":2937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24019110,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":23463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'2911","reported_seq":"3484","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.084194+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:14.084194+0000","last_peered":"2021-01-15T18:42:14.084194+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:14.084194+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2911,"ondisk_log_size":2911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23789734,"num_objects":2905,"num_object_clones":0,"num_object_copies":11620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2909,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":2072576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1753}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3102","reported_seq":"3167","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:13.990745+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:13.990745+0000","last_peered":"2021-01-15T18:42:13.990745+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:13.990745+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3102,"ondisk_log_size":3102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25362598,"num_objects":3097,"num_object_clones":0,"num_object_copies":12388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3100,"num_write_kb":24780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3135","reported_seq":"3195","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:14.316160+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:14.316160+0000","last_peered":"2021-01-15T18:42:14.316160+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:14.316160+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3135,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25600332,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3132,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1641}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'2939","reported_seq":"3020","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:13.115456+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:13.115456+0000","last_peered":"2021-01-15T18:42:13.115456+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:13.115456+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2939,"ondisk_log_size":2939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24051878,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2937,"num_write_kb":23490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389155092,"num_objects":47518,"num_object_clones":0,"num_object_copies":190072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47545,"num_write_kb":380150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4693,"num_bytes_recovered":38166528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":707457024,"data_stored":707457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47572,"ondisk_log_size":47572,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215137,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287264,"kb_used_data":191384,"kb_used_omap":0,"kb_used_meta":95871,"kb_avail":94084576,"statfs":{"total":96636764160,"available":96342605824,"internally_reserved":0,"allocated":195977216,"data_stored":195782094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":98171956},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":0.94999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":0.53700000000000003},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.256}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280420,"kb_used_data":189404,"kb_used_omap":0,"kb_used_meta":91008,"kb_avail":94091420,"statfs":{"total":96636764160,"available":96349614080,"internally_reserved":0,"allocated":193949696,"data_stored":193754574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93192192},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.6759999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":2.0630000000000002},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":1.843}]}]},{"osd":0,"up_from":22,"seq":94489280547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295432,"kb_used_data":191360,"kb_used_omap":0,"kb_used_meta":104063,"kb_avail":94076408,"statfs":{"total":96636764160,"available":96334241792,"internally_reserved":0,"allocated":195952640,"data_stored":195757518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":106560564},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,4,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":8.6929999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.76600000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.9939999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.1200000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":30.213000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188172,"kb_used_data":117828,"kb_used_omap":0,"kb_used_meta":70335,"kb_avail":94183668,"statfs":{"total":96636764160,"available":96444076032,"internally_reserved":0,"allocated":120655872,"data_stored":120460750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":72023092},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":1.252},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":6.6130000000000004},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":6.915}]}]}]}} 2021-01-15T18:42:17.045 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 137 20 48646 48626 2.7721 2.05469 0.0456013 0.0563473 2021-01-15T18:42:18.046 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 138 20 48986 48966 2.77123 2.65625 0.0864737 0.0563677 2021-01-15T18:42:19.047 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 139 20 49240 49220 2.76557 1.98438 0.0108215 0.0563654 2021-01-15T18:42:19.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:20.047 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:42:20.047775+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0566073 2021-01-15T18:42:20.048 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:42:20.048 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 140 20 49485 49465 2.75948 1.91406 0.0635923 0.0566073 2021-01-15T18:42:20.778 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:20.800 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:20.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2021-01-15T18:42:20.407468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":403335444,"num_objects":49249,"num_object_clones":0,"num_object_copies":196996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49276,"num_write_kb":393998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4800,"num_bytes_recovered":39043072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49303,"ondisk_log_size":49303,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1120460,"kb_used_data":720876,"kb_used_omap":2,"kb_used_meta":399549,"kb_avail":376366900,"statfs":{"total":386547056640,"available":385399705600,"internally_reserved":0,"allocated":738177024,"data_stored":737396536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":409138332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,5,15,29,29],"upper_bound":128},"perf_stat":{"commit_latency_ms":80,"apply_latency_ms":80,"commit_latency_ns":80000000,"apply_latency_ns":80000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36290560,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4430,"num_write_kb":35440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":2383872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005349"},"pg_stats":[{"pgid":"1.f","version":"36'2960","reported_seq":"3006","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.808599+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:18.808599+0000","last_peered":"2021-01-15T18:42:18.808599+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:18.808599+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2960,"ondisk_log_size":2960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248320,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2960,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3035","reported_seq":"5955","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.602561+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:19.602561+0000","last_peered":"2021-01-15T18:42:19.602561+0000","last_clean":"2021-01-15T18:42:19.602561+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:19.602561+0000","last_undegraded":"2021-01-15T18:42:19.602561+0000","last_fullsized":"2021-01-15T18:42:19.602561+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3035,"ondisk_log_size":3035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24821926,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":24252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3040","reported_seq":"3119","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.120376+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:18.120376+0000","last_peered":"2021-01-15T18:42:18.120376+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:18.120376+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3040,"ondisk_log_size":3040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24875091,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3039,"num_write_kb":24298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3126","reported_seq":"3172","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.756031+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:18.756031+0000","last_peered":"2021-01-15T18:42:18.756031+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:18.756031+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3126,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25608192,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3126,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3160","reported_seq":"3748","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.418612+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:19.418612+0000","last_peered":"2021-01-15T18:42:19.418612+0000","last_clean":"2021-01-15T18:42:19.418612+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:19.418612+0000","last_undegraded":"2021-01-15T18:42:19.418612+0000","last_fullsized":"2021-01-15T18:42:19.418612+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3160,"ondisk_log_size":3160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25837734,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3063","reported_seq":"3270","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.436175+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:19.436175+0000","last_peered":"2021-01-15T18:42:19.436175+0000","last_clean":"2021-01-15T18:42:19.436175+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:19.436175+0000","last_undegraded":"2021-01-15T18:42:19.436175+0000","last_fullsized":"2021-01-15T18:42:19.436175+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3063,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25059494,"num_objects":3060,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3049","reported_seq":"3102","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:19.373039+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:19.373039+0000","last_peered":"2021-01-15T18:42:19.373039+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:19.373039+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3049,"ondisk_log_size":3049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24936614,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1653}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3087","reported_seq":"3161","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.223933+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:18.223933+0000","last_peered":"2021-01-15T18:42:18.223933+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:18.223933+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3087,"ondisk_log_size":3087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25276499,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3086,"num_write_kb":24685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3153","reported_seq":"3195","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.766244+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:18.766244+0000","last_peered":"2021-01-15T18:42:18.766244+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:18.766244+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3153,"ondisk_log_size":3153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25817171,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3113","reported_seq":"6050","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:18.736977+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:18.736977+0000","last_peered":"2021-01-15T18:42:18.736977+0000","last_clean":"2021-01-15T18:42:18.736977+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:18.736977+0000","last_undegraded":"2021-01-15T18:42:18.736977+0000","last_fullsized":"2021-01-15T18:42:18.736977+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3113,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25448697,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":24860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3031","reported_seq":"3088","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:19.352575+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:19.352575+0000","last_peered":"2021-01-15T18:42:19.352575+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:19.352575+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24797350,"num_objects":3028,"num_object_clones":0,"num_object_copies":12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3034","reported_seq":"3098","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.565396+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:18.565396+0000","last_peered":"2021-01-15T18:42:18.565396+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:18.565396+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3034,"ondisk_log_size":3034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24813734,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3032,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3009","reported_seq":"3795","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.686969+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:18.686969+0000","last_peered":"2021-01-15T18:42:18.686969+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:18.686969+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3009,"ondisk_log_size":3009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24592550,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":24028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3192","reported_seq":"3257","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.476563+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:18.476563+0000","last_peered":"2021-01-15T18:42:18.476563+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:18.476563+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3192,"ondisk_log_size":3192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26099878,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":25500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3223","reported_seq":"3281","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.760496+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:18.760496+0000","last_peered":"2021-01-15T18:42:18.760496+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:18.760496+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3223,"ondisk_log_size":3223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26321228,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3028","reported_seq":"3109","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.011056+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:18.011056+0000","last_peered":"2021-01-15T18:42:18.011056+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:18.011056+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3028,"ondisk_log_size":3028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24780966,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":403335444,"num_objects":49249,"num_object_clones":0,"num_object_copies":196996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49276,"num_write_kb":393998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4800,"num_bytes_recovered":39043072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738680832,"data_stored":738680832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49303,"ondisk_log_size":49303,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303892,"kb_used_data":197260,"kb_used_omap":0,"kb_used_meta":106623,"kb_avail":94067948,"statfs":{"total":96636764160,"available":96325578752,"internally_reserved":0,"allocated":201994240,"data_stored":201799118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":109182004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,0,5,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":16.355},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":16.629000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.758},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.5569999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297340,"kb_used_data":195444,"kb_used_omap":0,"kb_used_meta":101888,"kb_avail":94074500,"statfs":{"total":96636764160,"available":96332288000,"internally_reserved":0,"allocated":200134656,"data_stored":199939534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104333312},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,10,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.0269999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":2.6219999999999999},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":2.73}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.143},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":5.9870000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295732,"kb_used_data":197228,"kb_used_omap":0,"kb_used_meta":98495,"kb_avail":94076108,"statfs":{"total":96636764160,"available":96333934592,"internally_reserved":0,"allocated":201961472,"data_stored":201766350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":100858932},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,11,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.80000000000000004},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":12.925000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.56499999999999995},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.0679999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.1890000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":28.07}]}]},{"osd":1,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":223496,"kb_used_data":130944,"kb_used_omap":0,"kb_used_meta":92543,"kb_avail":94148344,"statfs":{"total":96636764160,"available":96407904256,"internally_reserved":0,"allocated":134086656,"data_stored":133891534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":94764084},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":5.6459999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":2.7669999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":5.3540000000000001},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.516}]}]}]}} 2021-01-15T18:42:20.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:21.047 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 141 20 49816 49796 2.75824 2.58594 0.0388584 0.0566352 2021-01-15T18:42:22.049 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 142 20 50142 50122 2.75672 2.54688 0.0409826 0.0566665 2021-01-15T18:42:22.268 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:22.269 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16605/196996 objects degraded (8.429%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49249,"data_bytes":403335444,"bytes_used":1147351040,"bytes_avail":385399705600,"bytes_total":386547056640,"degraded_objects":16605,"degraded_total":196996,"degraded_ratio":0.084291051594956243,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":198567,"recovering_keys_per_sec":0,"num_objects_recovered":291,"num_bytes_recovered":2383872,"num_keys_recovered":0,"write_bytes_sec":3022865,"read_op_per_sec":0,"write_op_per_sec":369},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:22.320 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:22.321 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:23.049 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 143 20 50470 50450 2.75536 2.5625 0.0557556 0.0566922 2021-01-15T18:42:23.586 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:23.627 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:23.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2021-01-15T18:42:22.408817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":403335444,"num_objects":49249,"num_object_clones":0,"num_object_copies":196996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49276,"num_write_kb":393998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4800,"num_bytes_recovered":39043072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49303,"ondisk_log_size":49303,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1120460,"kb_used_data":720876,"kb_used_omap":2,"kb_used_meta":399549,"kb_avail":376366900,"statfs":{"total":386547056640,"available":385399705600,"internally_reserved":0,"allocated":738177024,"data_stored":737396536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":409138332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,5,15,29,29],"upper_bound":128},"perf_stat":{"commit_latency_ms":80,"apply_latency_ms":80,"commit_latency_ns":80000000,"apply_latency_ns":80000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":1712128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006169"},"pg_stats":[{"pgid":"1.f","version":"36'2960","reported_seq":"3006","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.808599+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:18.808599+0000","last_peered":"2021-01-15T18:42:18.808599+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:18.808599+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":2960,"ondisk_log_size":2960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248320,"num_objects":2960,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2960,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3035","reported_seq":"5955","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.602561+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:19.602561+0000","last_peered":"2021-01-15T18:42:19.602561+0000","last_clean":"2021-01-15T18:42:19.602561+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:19.602561+0000","last_undegraded":"2021-01-15T18:42:19.602561+0000","last_fullsized":"2021-01-15T18:42:19.602561+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3035,"ondisk_log_size":3035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24821926,"num_objects":3031,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":24252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3040","reported_seq":"3119","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.120376+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:18.120376+0000","last_peered":"2021-01-15T18:42:18.120376+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:18.120376+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3040,"ondisk_log_size":3040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24875091,"num_objects":3037,"num_object_clones":0,"num_object_copies":12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3039,"num_write_kb":24298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1621}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3126","reported_seq":"3172","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.756031+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:18.756031+0000","last_peered":"2021-01-15T18:42:18.756031+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:18.756031+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3126,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25608192,"num_objects":3126,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3126,"num_write_kb":25008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3160","reported_seq":"3748","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.418612+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:19.418612+0000","last_peered":"2021-01-15T18:42:19.418612+0000","last_clean":"2021-01-15T18:42:19.418612+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:19.418612+0000","last_undegraded":"2021-01-15T18:42:19.418612+0000","last_fullsized":"2021-01-15T18:42:19.418612+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3160,"ondisk_log_size":3160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25837734,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3063","reported_seq":"3270","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:19.436175+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:19.436175+0000","last_peered":"2021-01-15T18:42:19.436175+0000","last_clean":"2021-01-15T18:42:19.436175+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:19.436175+0000","last_undegraded":"2021-01-15T18:42:19.436175+0000","last_fullsized":"2021-01-15T18:42:19.436175+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3063,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25059494,"num_objects":3060,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":24479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3049","reported_seq":"3102","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:19.373039+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:19.373039+0000","last_peered":"2021-01-15T18:42:19.373039+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:19.373039+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3049,"ondisk_log_size":3049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24936614,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":24364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1653}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3087","reported_seq":"3161","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.223933+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:18.223933+0000","last_peered":"2021-01-15T18:42:18.223933+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:18.223933+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3087,"ondisk_log_size":3087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25276499,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3086,"num_write_kb":24685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3153","reported_seq":"3195","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.766244+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:18.766244+0000","last_peered":"2021-01-15T18:42:18.766244+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:18.766244+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3153,"ondisk_log_size":3153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25817171,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3113","reported_seq":"6050","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:18.736977+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:18.736977+0000","last_peered":"2021-01-15T18:42:18.736977+0000","last_clean":"2021-01-15T18:42:18.736977+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:18.736977+0000","last_undegraded":"2021-01-15T18:42:18.736977+0000","last_fullsized":"2021-01-15T18:42:18.736977+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3113,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25448697,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":24860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3031","reported_seq":"3088","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:19.352575+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:19.352575+0000","last_peered":"2021-01-15T18:42:19.352575+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:19.352575+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24797350,"num_objects":3028,"num_object_clones":0,"num_object_copies":12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3034","reported_seq":"3098","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.565396+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:18.565396+0000","last_peered":"2021-01-15T18:42:18.565396+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:18.565396+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3034,"ondisk_log_size":3034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24813734,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3032,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3009","reported_seq":"3795","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.686969+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:18.686969+0000","last_peered":"2021-01-15T18:42:18.686969+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:18.686969+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3009,"ondisk_log_size":3009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24592550,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":24028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":362,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3192","reported_seq":"3257","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.476563+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:18.476563+0000","last_peered":"2021-01-15T18:42:18.476563+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:18.476563+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3192,"ondisk_log_size":3192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26099878,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":25500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3223","reported_seq":"3281","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.760496+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:18.760496+0000","last_peered":"2021-01-15T18:42:18.760496+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:18.760496+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3223,"ondisk_log_size":3223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26321228,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3028","reported_seq":"3109","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:18.011056+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:18.011056+0000","last_peered":"2021-01-15T18:42:18.011056+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:18.011056+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3028,"ondisk_log_size":3028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24780966,"num_objects":3026,"num_object_clones":0,"num_object_copies":12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":24202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":403335444,"num_objects":49249,"num_object_clones":0,"num_object_copies":196996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49276,"num_write_kb":393998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4800,"num_bytes_recovered":39043072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738680832,"data_stored":738680832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49303,"ondisk_log_size":49303,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215138,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303892,"kb_used_data":197260,"kb_used_omap":0,"kb_used_meta":106623,"kb_avail":94067948,"statfs":{"total":96636764160,"available":96325578752,"internally_reserved":0,"allocated":201994240,"data_stored":201799118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":109182004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,0,5,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":16.355},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":16.629000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":1.758},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.5569999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297340,"kb_used_data":195444,"kb_used_omap":0,"kb_used_meta":101888,"kb_avail":94074500,"statfs":{"total":96636764160,"available":96332288000,"internally_reserved":0,"allocated":200134656,"data_stored":199939534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104333312},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,10,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.0269999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":2.6219999999999999},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":2.73}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.143},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":5.9870000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295732,"kb_used_data":197228,"kb_used_omap":0,"kb_used_meta":98495,"kb_avail":94076108,"statfs":{"total":96636764160,"available":96333934592,"internally_reserved":0,"allocated":201961472,"data_stored":201766350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":100858932},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,11,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.80000000000000004},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":12.925000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.56499999999999995},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.0679999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.1890000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":28.07}]}]},{"osd":1,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":223496,"kb_used_data":130944,"kb_used_omap":0,"kb_used_meta":92543,"kb_avail":94148344,"statfs":{"total":96636764160,"available":96407904256,"internally_reserved":0,"allocated":134086656,"data_stored":133891534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":94764084},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":5.6459999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":2.7669999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":5.3540000000000001},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.516}]}]}]}} 2021-01-15T18:42:24.049 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 144 20 50801 50781 2.75419 2.58594 0.07972 0.0567181 2021-01-15T18:42:25.051 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 145 20 51136 51116 2.75321 2.61719 0.044714 0.0567368 2021-01-15T18:42:26.051 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 146 20 51468 51448 2.75212 2.59375 0.0484331 0.0567601 2021-01-15T18:42:26.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:27.051 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 147 20 51784 51764 2.75019 2.46875 0.0789851 0.0568054 2021-01-15T18:42:27.572 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:27.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-01-15T18:42:26.409937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416573716,"num_objects":50865,"num_object_clones":0,"num_object_copies":203460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50892,"num_write_kb":406926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4883,"num_bytes_recovered":39723008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50919,"ondisk_log_size":50919,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1130456,"kb_used_data":744820,"kb_used_omap":2,"kb_used_meta":385597,"kb_avail":376356904,"statfs":{"total":386547056640,"available":385389469696,"internally_reserved":0,"allocated":762695680,"data_stored":761915192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":394851484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,12,7,20,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":86,"apply_latency_ms":86,"commit_latency_ns":86000000,"apply_latency_ns":86000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27418624,"num_objects":3347,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3347,"num_write_kb":26776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004654"},"pg_stats":[{"pgid":"1.f","version":"36'3072","reported_seq":"3118","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.311095+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:24.311095+0000","last_peered":"2021-01-15T18:42:24.311095+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:24.311095+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3072,"ondisk_log_size":3072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165824,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3142","reported_seq":"6062","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.598768+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:24.598768+0000","last_peered":"2021-01-15T18:42:24.598768+0000","last_clean":"2021-01-15T18:42:24.598768+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:24.598768+0000","last_undegraded":"2021-01-15T18:42:24.598768+0000","last_fullsized":"2021-01-15T18:42:24.598768+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3142,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25698470,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3140,"num_write_kb":25108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3125","reported_seq":"3202","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:23.175915+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:23.175915+0000","last_peered":"2021-01-15T18:42:23.175915+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:23.175915+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3125,"ondisk_log_size":3125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25571411,"num_objects":3122,"num_object_clones":0,"num_object_copies":12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":24978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1706}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3231","reported_seq":"3277","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.370683+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:24.370683+0000","last_peered":"2021-01-15T18:42:24.370683+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:24.370683+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3231,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26468352,"num_objects":3231,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":25848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3267","reported_seq":"3855","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.525402+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:24.525402+0000","last_peered":"2021-01-15T18:42:24.525402+0000","last_clean":"2021-01-15T18:42:24.525402+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:24.525402+0000","last_undegraded":"2021-01-15T18:42:24.525402+0000","last_fullsized":"2021-01-15T18:42:24.525402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3267,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26714278,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3164","reported_seq":"3371","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.523347+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:24.523347+0000","last_peered":"2021-01-15T18:42:24.523347+0000","last_clean":"2021-01-15T18:42:24.523347+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:24.523347+0000","last_undegraded":"2021-01-15T18:42:24.523347+0000","last_fullsized":"2021-01-15T18:42:24.523347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3164,"ondisk_log_size":3164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25886886,"num_objects":3161,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":25287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3154","reported_seq":"3207","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.371983+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:24.371983+0000","last_peered":"2021-01-15T18:42:24.371983+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:24.371983+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3154,"ondisk_log_size":3154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25796774,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3176","reported_seq":"3250","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:23.109014+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:23.109014+0000","last_peered":"2021-01-15T18:42:23.109014+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:23.109014+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3176,"ondisk_log_size":3176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26005587,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":25397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1723}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3236","reported_seq":"3277","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.380752+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:24.380752+0000","last_peered":"2021-01-15T18:42:24.380752+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:24.380752+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3236,"ondisk_log_size":3236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26497107,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":25877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3224","reported_seq":"6161","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.277149+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:24.277149+0000","last_peered":"2021-01-15T18:42:24.277149+0000","last_clean":"2021-01-15T18:42:24.277149+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:24.277149+0000","last_undegraded":"2021-01-15T18:42:24.277149+0000","last_fullsized":"2021-01-15T18:42:24.277149+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3224,"ondisk_log_size":3224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26358009,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":25748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3135","reported_seq":"3193","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.301429+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:24.301429+0000","last_peered":"2021-01-15T18:42:24.301429+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:24.301429+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3135,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25649318,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3136","reported_seq":"3200","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.013528+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:24.013528+0000","last_peered":"2021-01-15T18:42:24.013528+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:24.013528+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3136,"ondisk_log_size":3136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25649318,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3134,"num_write_kb":25055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3106","reported_seq":"4059","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.058628+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:24.058628+0000","last_peered":"2021-01-15T18:42:24.058628+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:24.058628+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3106,"ondisk_log_size":3106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25387174,"num_objects":3100,"num_object_clones":0,"num_object_copies":12400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3302","reported_seq":"3367","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:23.924699+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:23.924699+0000","last_peered":"2021-01-15T18:42:23.924699+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:23.924699+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3302,"ondisk_log_size":3302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27000998,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":26380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3327","reported_seq":"3387","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.313624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:24.313624+0000","last_peered":"2021-01-15T18:42:24.313624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:24.313624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3327,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27173196,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3122","reported_seq":"3203","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:23.131535+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:23.131535+0000","last_peered":"2021-01-15T18:42:23.131535+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:23.131535+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3122,"ondisk_log_size":3122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25551014,"num_objects":3120,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":24954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1738}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416573716,"num_objects":50865,"num_object_clones":0,"num_object_copies":203460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50892,"num_write_kb":406926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4883,"num_bytes_recovered":39723008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":765087744,"data_stored":765087744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50919,"ondisk_log_size":50919,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300964,"kb_used_data":203740,"kb_used_omap":0,"kb_used_meta":97215,"kb_avail":94070876,"statfs":{"total":96636764160,"available":96328577024,"internally_reserved":0,"allocated":208629760,"data_stored":208434638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":99548212},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,0,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.3849999999999998},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":4.4850000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":5.718}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":7.9199999999999999},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.76400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297684,"kb_used_data":201352,"kb_used_omap":0,"kb_used_meta":96320,"kb_avail":94074156,"statfs":{"total":96636764160,"available":96331935744,"internally_reserved":0,"allocated":206184448,"data_stored":205989326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98631680},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.4099999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.91100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":4.7949999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":0.58999999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311060,"kb_used_data":202188,"kb_used_omap":0,"kb_used_meta":108863,"kb_avail":94060780,"statfs":{"total":96636764160,"available":96318238720,"internally_reserved":0,"allocated":207040512,"data_stored":206845390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":111475764},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2,10,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.851},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":5.5259999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":1.956},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.2839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.9400000000000004},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":6.1200000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220748,"kb_used_data":137540,"kb_used_omap":0,"kb_used_meta":83199,"kb_avail":94151092,"statfs":{"total":96636764160,"available":96410718208,"internally_reserved":0,"allocated":140840960,"data_stored":140645838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":85195828},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":1.506},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.238},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.706},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.8080000000000001}]}]}]}} 2021-01-15T18:42:27.657 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:28.055 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 148 20 52077 52057 2.74702 2.28906 0.0756269 0.0568611 2021-01-15T18:42:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:28.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16522/204672 objects degraded (8.072%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":199,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51168,"data_bytes":419055892,"bytes_used":1175810048,"bytes_avail":385371246592,"bytes_total":386547056640,"degraded_objects":16522,"degraded_total":204672,"degraded_ratio":0.080724280800500314,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":129661,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"write_bytes_sec":2297063,"read_op_per_sec":0,"write_op_per_sec":280},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-01-15T18:41:30.390318+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:29.030 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:29.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:29.054 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 149 20 52393 52373 2.74515 2.46875 0.0643885 0.0569004 2021-01-15T18:42:29.901 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:29.922 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:29.971 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2021-01-15T18:42:28.410446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":419055892,"num_objects":51168,"num_object_clones":0,"num_object_copies":204672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51195,"num_write_kb":409350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4883,"num_bytes_recovered":39723008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51222,"ondisk_log_size":51222,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1148252,"kb_used_data":751484,"kb_used_omap":2,"kb_used_meta":396733,"kb_avail":376339108,"statfs":{"total":386547056640,"available":385371246592,"internally_reserved":0,"allocated":769519616,"data_stored":768739128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":406254748},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,11,7,25,23,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":88,"apply_latency_ms":88,"commit_latency_ns":88000000,"apply_latency_ns":88000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27574272,"num_objects":3366,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":26928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004141"},"pg_stats":[{"pgid":"1.f","version":"36'3072","reported_seq":"3118","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.311095+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:24.311095+0000","last_peered":"2021-01-15T18:42:24.311095+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:24.311095+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3072,"ondisk_log_size":3072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165824,"num_objects":3072,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1718}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3142","reported_seq":"6062","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.598768+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:24.598768+0000","last_peered":"2021-01-15T18:42:24.598768+0000","last_clean":"2021-01-15T18:42:24.598768+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:24.598768+0000","last_undegraded":"2021-01-15T18:42:24.598768+0000","last_fullsized":"2021-01-15T18:42:24.598768+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3142,"ondisk_log_size":3142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25698470,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3140,"num_write_kb":25108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3225","reported_seq":"3303","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.198726+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:28.198726+0000","last_peered":"2021-01-15T18:42:28.198726+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:28.198726+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3225,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26390611,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3231","reported_seq":"3277","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.370683+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:24.370683+0000","last_peered":"2021-01-15T18:42:24.370683+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:24.370683+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3231,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26468352,"num_objects":3231,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":25848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3267","reported_seq":"3855","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.525402+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:24.525402+0000","last_peered":"2021-01-15T18:42:24.525402+0000","last_clean":"2021-01-15T18:42:24.525402+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:24.525402+0000","last_undegraded":"2021-01-15T18:42:24.525402+0000","last_fullsized":"2021-01-15T18:42:24.525402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3267,"ondisk_log_size":3267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26714278,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":26100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3164","reported_seq":"3371","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.523347+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:24.523347+0000","last_peered":"2021-01-15T18:42:24.523347+0000","last_clean":"2021-01-15T18:42:24.523347+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:24.523347+0000","last_undegraded":"2021-01-15T18:42:24.523347+0000","last_fullsized":"2021-01-15T18:42:24.523347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3164,"ondisk_log_size":3164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25886886,"num_objects":3161,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":25287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3154","reported_seq":"3207","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.371983+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:24.371983+0000","last_peered":"2021-01-15T18:42:24.371983+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:24.371983+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3154,"ondisk_log_size":3154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25796774,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3152,"num_write_kb":25204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3280","reported_seq":"3354","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.232654+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:28.232654+0000","last_peered":"2021-01-15T18:42:28.232654+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:28.232654+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3280,"ondisk_log_size":3280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26857555,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3279,"num_write_kb":26229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3236","reported_seq":"3277","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.380752+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:24.380752+0000","last_peered":"2021-01-15T18:42:24.380752+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:24.380752+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3236,"ondisk_log_size":3236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26497107,"num_objects":3235,"num_object_clones":0,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":25877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3224","reported_seq":"6161","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:24.277149+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:24.277149+0000","last_peered":"2021-01-15T18:42:24.277149+0000","last_clean":"2021-01-15T18:42:24.277149+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:24.277149+0000","last_undegraded":"2021-01-15T18:42:24.277149+0000","last_fullsized":"2021-01-15T18:42:24.277149+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3224,"ondisk_log_size":3224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26358009,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":25748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3135","reported_seq":"3193","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.301429+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:24.301429+0000","last_peered":"2021-01-15T18:42:24.301429+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:24.301429+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3135,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25649318,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3136","reported_seq":"3200","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.013528+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:24.013528+0000","last_peered":"2021-01-15T18:42:24.013528+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:24.013528+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3136,"ondisk_log_size":3136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25649318,"num_objects":3132,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3134,"num_write_kb":25055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3106","reported_seq":"4059","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.058628+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:24.058628+0000","last_peered":"2021-01-15T18:42:24.058628+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:24.058628+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3106,"ondisk_log_size":3106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25387174,"num_objects":3100,"num_object_clones":0,"num_object_copies":12400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3302","reported_seq":"3367","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:23.924699+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:23.924699+0000","last_peered":"2021-01-15T18:42:23.924699+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:23.924699+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3302,"ondisk_log_size":3302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27000998,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":26380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3327","reported_seq":"3387","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:24.313624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:24.313624+0000","last_peered":"2021-01-15T18:42:24.313624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:24.313624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3327,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27173196,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1833}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3221","reported_seq":"3302","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.200903+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:28.200903+0000","last_peered":"2021-01-15T18:42:28.200903+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:28.200903+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3221,"ondisk_log_size":3221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26362022,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":419055892,"num_objects":51168,"num_object_clones":0,"num_object_copies":204672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51195,"num_write_kb":409350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4883,"num_bytes_recovered":39723008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":771661824,"data_stored":771661824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51222,"ondisk_log_size":51222,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215139,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300964,"kb_used_data":203740,"kb_used_omap":0,"kb_used_meta":97215,"kb_avail":94070876,"statfs":{"total":96636764160,"available":96328577024,"internally_reserved":0,"allocated":208629760,"data_stored":208434638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":99548212},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,0,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.3849999999999998},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":4.4850000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":5.718}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":7.9199999999999999},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.76400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315480,"kb_used_data":208016,"kb_used_omap":0,"kb_used_meta":107456,"kb_avail":94056360,"statfs":{"total":96636764160,"available":96313712640,"internally_reserved":0,"allocated":213008384,"data_stored":212813262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":110034944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":2.0190000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":1.2769999999999999},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":1.8140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":2.1360000000000001},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":2.399}]}]},{"osd":0,"up_from":22,"seq":94489280549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311060,"kb_used_data":202188,"kb_used_omap":0,"kb_used_meta":108863,"kb_avail":94060780,"statfs":{"total":96636764160,"available":96318238720,"internally_reserved":0,"allocated":207040512,"data_stored":206845390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":111475764},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2,10,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.851},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":5.5259999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":1.956},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.2839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.9400000000000004},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":6.1200000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220748,"kb_used_data":137540,"kb_used_omap":0,"kb_used_meta":83199,"kb_avail":94151092,"statfs":{"total":96636764160,"available":96410718208,"internally_reserved":0,"allocated":140840960,"data_stored":140645838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":85195828},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":1.506},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.238},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.706},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.8080000000000001}]}]}]}} 2021-01-15T18:42:30.054 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 150 20 52689 52669 2.74227 2.3125 0.0724482 0.0569636 2021-01-15T18:42:31.054 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 151 20 52905 52885 2.73528 1.6875 0.0130059 0.0569998 2021-01-15T18:42:32.054 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 152 20 53108 53088 2.72772 1.58594 0.0608406 0.057265 2021-01-15T18:42:32.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:33.054 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 153 20 53435 53415 2.72659 2.55469 0.0711756 0.0572903 2021-01-15T18:42:34.055 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 154 20 53727 53707 2.72368 2.28125 0.0785385 0.057353 2021-01-15T18:42:34.139 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:34.170 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:34.210 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2021-01-15T18:42:32.411526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429631764,"num_objects":52459,"num_object_clones":0,"num_object_copies":209836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52486,"num_write_kb":419678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4977,"num_bytes_recovered":40493056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52513,"ondisk_log_size":52513,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1184764,"kb_used_data":771184,"kb_used_omap":2,"kb_used_meta":413565,"kb_avail":376302596,"statfs":{"total":386547056640,"available":385333858304,"internally_reserved":0,"allocated":789692416,"data_stored":788911928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":423490716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,10,29,31,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":88,"apply_latency_ms":88,"commit_latency_ns":88000000,"apply_latency_ns":88000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26296320,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":25680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004058"},"pg_stats":[{"pgid":"1.f","version":"36'3178","reported_seq":"3224","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.351896+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:29.351896+0000","last_peered":"2021-01-15T18:42:29.351896+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:29.351896+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3178,"ondisk_log_size":3178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26034176,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3178,"num_write_kb":25424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3238","reported_seq":"6158","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:29.585461+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:29.585461+0000","last_peered":"2021-01-15T18:42:29.585461+0000","last_clean":"2021-01-15T18:42:29.585461+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:29.585461+0000","last_undegraded":"2021-01-15T18:42:29.585461+0000","last_fullsized":"2021-01-15T18:42:29.585461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3238,"ondisk_log_size":3238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26484902,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":25876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3225","reported_seq":"3303","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.198726+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:28.198726+0000","last_peered":"2021-01-15T18:42:28.198726+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:28.198726+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3225,"ondisk_log_size":3225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26390611,"num_objects":3222,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":25778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3334","reported_seq":"3380","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.285240+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:29.285240+0000","last_peered":"2021-01-15T18:42:29.285240+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:29.285240+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3334,"ondisk_log_size":3334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27312128,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3334,"num_write_kb":26672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3367","reported_seq":"3954","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:29.653250+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:29.653250+0000","last_peered":"2021-01-15T18:42:29.653250+0000","last_clean":"2021-01-15T18:42:29.653250+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:29.653250+0000","last_undegraded":"2021-01-15T18:42:29.653250+0000","last_fullsized":"2021-01-15T18:42:29.653250+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3367,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27533478,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3259","reported_seq":"3466","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:29.607192+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:29.607192+0000","last_peered":"2021-01-15T18:42:29.607192+0000","last_clean":"2021-01-15T18:42:29.607192+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:29.607192+0000","last_undegraded":"2021-01-15T18:42:29.607192+0000","last_fullsized":"2021-01-15T18:42:29.607192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3259,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26665126,"num_objects":3256,"num_object_clones":0,"num_object_copies":13024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3254","reported_seq":"3307","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.400557+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:29.400557+0000","last_peered":"2021-01-15T18:42:29.400557+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:29.400557+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3254,"ondisk_log_size":3254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26615974,"num_objects":3250,"num_object_clones":0,"num_object_copies":13000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":26004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3280","reported_seq":"3354","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.232654+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:28.232654+0000","last_peered":"2021-01-15T18:42:28.232654+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:28.232654+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3280,"ondisk_log_size":3280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26857555,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3279,"num_write_kb":26229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3341","reported_seq":"3383","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.276011+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:29.276011+0000","last_peered":"2021-01-15T18:42:29.276011+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:29.276011+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3341,"ondisk_log_size":3341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27357267,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3340,"num_write_kb":26717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1904}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3318","reported_seq":"6255","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:29.402378+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:29.402378+0000","last_peered":"2021-01-15T18:42:29.402378+0000","last_clean":"2021-01-15T18:42:29.402378+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:29.402378+0000","last_undegraded":"2021-01-15T18:42:29.402378+0000","last_fullsized":"2021-01-15T18:42:29.402378+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3318,"ondisk_log_size":3318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27128057,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3315,"num_write_kb":26500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3222","reported_seq":"3280","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.372209+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:29.372209+0000","last_peered":"2021-01-15T18:42:29.372209+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:29.372209+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3222,"ondisk_log_size":3222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26362022,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3237","reported_seq":"3301","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.025005+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:29.025005+0000","last_peered":"2021-01-15T18:42:29.025005+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:29.025005+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3237,"ondisk_log_size":3237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26476710,"num_objects":3233,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":25863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1862}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3198","reported_seq":"4338","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.084035+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:29.084035+0000","last_peered":"2021-01-15T18:42:29.084035+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:29.084035+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3198,"ondisk_log_size":3198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26140838,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":868,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3196,"num_write_kb":25540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":4399104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3409","reported_seq":"3474","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.089953+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:29.089953+0000","last_peered":"2021-01-15T18:42:29.089953+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:29.089953+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3409,"ondisk_log_size":3409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27877542,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":27236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3432","reported_seq":"3492","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:29.194463+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:29.194463+0000","last_peered":"2021-01-15T18:42:29.194463+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:29.194463+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3432,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28033356,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":27395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1938}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3221","reported_seq":"3302","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:28.200903+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:28.200903+0000","last_peered":"2021-01-15T18:42:28.200903+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:28.200903+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3221,"ondisk_log_size":3221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26362022,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":429631764,"num_objects":52459,"num_object_clones":0,"num_object_copies":209836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52486,"num_write_kb":419678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4977,"num_bytes_recovered":40493056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":791515136,"data_stored":791515136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52513,"ondisk_log_size":52513,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215140,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318412,"kb_used_data":210192,"kb_used_omap":0,"kb_used_meta":108223,"kb_avail":94053428,"statfs":{"total":96636764160,"available":96310710272,"internally_reserved":0,"allocated":215236608,"data_stored":215041486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":110820404},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,8,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":6.4189999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":6.6159999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:41:33 2021","interfaces":[{"interface":"back","average":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":7.8700000000000001},"last":6.3099999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":8.2469999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":8.5679999999999996},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.7249999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315480,"kb_used_data":208016,"kb_used_omap":0,"kb_used_meta":107456,"kb_avail":94056360,"statfs":{"total":96636764160,"available":96313712640,"internally_reserved":0,"allocated":213008384,"data_stored":212813262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":110034944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":2.0190000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.286,"15min":2.286},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":1.2769999999999999},{"interface":"front","average":{"1min":2.069,"5min":2.069,"15min":2.069},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":1.8140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":2.1360000000000001},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":2.399}]}]},{"osd":0,"up_from":22,"seq":94489280550,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312280,"kb_used_data":208720,"kb_used_omap":0,"kb_used_meta":103551,"kb_avail":94059560,"statfs":{"total":96636764160,"available":96316989440,"internally_reserved":0,"allocated":213729280,"data_stored":213534158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":106036276},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:41:31 2021","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":1.5860000000000001},{"interface":"front","average":{"1min":2.093,"5min":2.093,"15min":2.093},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.337,"5min":17.337,"15min":17.337},"last":3.8860000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":1.637},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":8.4930000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":1.5249999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238592,"kb_used_data":144256,"kb_used_omap":0,"kb_used_meta":94335,"kb_avail":94133248,"statfs":{"total":96636764160,"available":96392445952,"internally_reserved":0,"allocated":147718144,"data_stored":147523022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":96599092},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":35.850999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.97299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.48099999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.57499999999999996}]}]}]}} 2021-01-15T18:42:34.212 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:35.055 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 155 20 54014 53994 2.72057 2.24219 0.0584013 0.0574129 2021-01-15T18:42:35.453 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:35.454 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16328/211816 objects degraded (7.709%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":205,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52954,"data_bytes":433686804,"bytes_used":1241686016,"bytes_avail":385305370624,"bytes_total":386547056640,"degraded_objects":16328,"degraded_total":211816,"degraded_ratio":0.0770857725573139,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":189048,"recovering_keys_per_sec":0,"num_objects_recovered":277,"num_bytes_recovered":2269184,"num_keys_recovered":0,"write_bytes_sec":2528604,"read_op_per_sec":0,"write_op_per_sec":308},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:35.481 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:35.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:36.056 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 156 20 54315 54295 2.71821 2.35156 0.0629527 0.0574655 2021-01-15T18:42:36.753 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:36.774 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:36.791 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2021-01-15T18:42:36.412487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440404244,"num_objects":53774,"num_object_clones":0,"num_object_copies":215096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53801,"num_write_kb":430198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5077,"num_bytes_recovered":41312256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53828,"ondisk_log_size":53828,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1220556,"kb_used_data":793148,"kb_used_omap":2,"kb_used_meta":427389,"kb_avail":376266804,"statfs":{"total":386547056640,"available":385297207296,"internally_reserved":0,"allocated":812183552,"data_stored":811403064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":437646492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,13,8,19,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32342016,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3948,"num_write_kb":31584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003156"},"pg_stats":[{"pgid":"1.f","version":"36'3254","reported_seq":"3300","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.277567+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:34.277567+0000","last_peered":"2021-01-15T18:42:34.277567+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:34.277567+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3254,"ondisk_log_size":3254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26656768,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3254,"num_write_kb":26032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3315","reported_seq":"6235","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:34.431361+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:34.431361+0000","last_peered":"2021-01-15T18:42:34.431361+0000","last_clean":"2021-01-15T18:42:34.431361+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:34.431361+0000","last_undegraded":"2021-01-15T18:42:34.431361+0000","last_fullsized":"2021-01-15T18:42:34.431361+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3315,"ondisk_log_size":3315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27115686,"num_objects":3311,"num_object_clones":0,"num_object_copies":13244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":26492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3296","reported_seq":"3375","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:33.032506+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:33.032506+0000","last_peered":"2021-01-15T18:42:33.032506+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:33.032506+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3296,"ondisk_log_size":3296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26972243,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1877}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3426","reported_seq":"3472","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.373044+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:34.373044+0000","last_peered":"2021-01-15T18:42:34.373044+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:34.373044+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3426,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28065792,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":27408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3448","reported_seq":"4036","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:34.543302+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:34.543302+0000","last_peered":"2021-01-15T18:42:34.543302+0000","last_clean":"2021-01-15T18:42:34.543302+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:34.543302+0000","last_undegraded":"2021-01-15T18:42:34.543302+0000","last_fullsized":"2021-01-15T18:42:34.543302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3448,"ondisk_log_size":3448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28197030,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3337","reported_seq":"3544","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:34.431818+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:34.431818+0000","last_peered":"2021-01-15T18:42:34.431818+0000","last_clean":"2021-01-15T18:42:34.431818+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:34.431818+0000","last_undegraded":"2021-01-15T18:42:34.431818+0000","last_fullsized":"2021-01-15T18:42:34.431818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3337,"ondisk_log_size":3337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27304102,"num_objects":3334,"num_object_clones":0,"num_object_copies":13336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":26671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3342","reported_seq":"3396","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.360285+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:34.360285+0000","last_peered":"2021-01-15T18:42:34.360285+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:34.360285+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3342,"ondisk_log_size":3342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27336870,"num_objects":3338,"num_object_clones":0,"num_object_copies":13352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3340,"num_write_kb":26708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3375","reported_seq":"3449","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:33.113786+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:33.113786+0000","last_peered":"2021-01-15T18:42:33.113786+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:33.113786+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3375,"ondisk_log_size":3375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27635795,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3374,"num_write_kb":26989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1922}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3421","reported_seq":"3463","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.180544+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:34.180544+0000","last_peered":"2021-01-15T18:42:34.180544+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:34.180544+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3421,"ondisk_log_size":3421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28012627,"num_objects":3420,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3420,"num_write_kb":27357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1984}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3403","reported_seq":"6340","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:34.267433+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:34.267433+0000","last_peered":"2021-01-15T18:42:34.267433+0000","last_clean":"2021-01-15T18:42:34.267433+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:34.267433+0000","last_undegraded":"2021-01-15T18:42:34.267433+0000","last_fullsized":"2021-01-15T18:42:34.267433+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3403,"ondisk_log_size":3403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27824377,"num_objects":3398,"num_object_clones":0,"num_object_copies":13592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":27180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3299","reported_seq":"3357","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.312274+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:34.312274+0000","last_peered":"2021-01-15T18:42:34.312274+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:34.312274+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3299,"ondisk_log_size":3299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26992806,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3297,"num_write_kb":26367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1915}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3319","reported_seq":"3381","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.108478+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:34.108478+0000","last_peered":"2021-01-15T18:42:34.108478+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:34.108478+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3319,"ondisk_log_size":3319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27148454,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":26519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3280","reported_seq":"4620","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.067598+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:34.067598+0000","last_peered":"2021-01-15T18:42:34.067598+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:34.067598+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3280,"ondisk_log_size":3280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26812582,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3278,"num_write_kb":26196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3498","reported_seq":"3563","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:33.983681+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:33.983681+0000","last_peered":"2021-01-15T18:42:33.983681+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:33.983681+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3498,"ondisk_log_size":3498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28606630,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3518","reported_seq":"3578","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:34.214626+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:34.214626+0000","last_peered":"2021-01-15T18:42:34.214626+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:34.214626+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3518,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28737868,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3297","reported_seq":"3377","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:33.090320+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:33.090320+0000","last_peered":"2021-01-15T18:42:33.090320+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:33.090320+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3297,"ondisk_log_size":3297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26984614,"num_objects":3295,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3295,"num_write_kb":26354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440404244,"num_objects":53774,"num_object_clones":0,"num_object_copies":215096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53801,"num_write_kb":430198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5077,"num_bytes_recovered":41312256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":813768704,"data_stored":813768704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53828,"ondisk_log_size":53828,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215141,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330012,"kb_used_data":215396,"kb_used_omap":0,"kb_used_meta":114623,"kb_avail":94041828,"statfs":{"total":96636764160,"available":96298831872,"internally_reserved":0,"allocated":220565504,"data_stored":220370382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":117374004},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":14.138999999999999},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":14.478999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":6.2930000000000001},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":6.2229999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":5.5350000000000001},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.6500000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331444,"kb_used_data":213420,"kb_used_omap":0,"kb_used_meta":118016,"kb_avail":94040396,"statfs":{"total":96636764160,"available":96297365504,"internally_reserved":0,"allocated":218542080,"data_stored":218346958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":120848384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.673},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":2.9769999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":5.2830000000000004},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":5.0090000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":3.0840000000000001},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":3.7149999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324136,"kb_used_data":214112,"kb_used_omap":0,"kb_used_meta":110015,"kb_avail":94047704,"statfs":{"total":96636764160,"available":96304848896,"internally_reserved":0,"allocated":219250688,"data_stored":219055566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":112655412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.54600000000000004},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":11.464}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":11.598000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":11.494999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":11.426}]}]},{"osd":1,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":234964,"kb_used_data":150220,"kb_used_omap":0,"kb_used_meta":84735,"kb_avail":94136876,"statfs":{"total":96636764160,"available":96396161024,"internally_reserved":0,"allocated":153825280,"data_stored":153630158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":86768692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.541,"5min":12.541,"15min":12.541},"last":3.9500000000000002},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2599999999999998,"5min":9.2599999999999998,"15min":9.2599999999999998},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":3.7010000000000001},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":3.7999999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:41:32 2021","interfaces":[{"interface":"back","average":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.1920000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":19.745000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.9060000000000001},{"interface":"front","average":{"1min":2.633,"5min":2.633,"15min":2.633},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":3.6779999999999999}]}]}]}} 2021-01-15T18:42:37.056 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 157 20 54621 54601 2.71612 2.39062 0.0402318 0.0575097 2021-01-15T18:42:38.056 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 158 20 54946 54926 2.715 2.53906 0.057649 0.0575297 2021-01-15T18:42:39.056 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 159 20 55262 55242 2.71345 2.46875 0.0610285 0.0575653 2021-01-15T18:42:39.794 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:40.059 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:42:40.059670+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0575964 2021-01-15T18:42:40.060 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:42:40.060 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 160 20 55584 55564 2.71216 2.51562 0.09271 0.0575964 2021-01-15T18:42:41.040 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:41.064 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 161 20 55884 55864 2.70979 2.34375 0.0711998 0.0576457 2021-01-15T18:42:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:41.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2021-01-15T18:42:40.413642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":453118228,"num_objects":55326,"num_object_clones":0,"num_object_copies":221304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55353,"num_write_kb":442614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5163,"num_bytes_recovered":42016768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55380,"ondisk_log_size":55380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1236364,"kb_used_data":818228,"kb_used_omap":2,"kb_used_meta":418109,"kb_avail":376250996,"statfs":{"total":386547056640,"available":385281019904,"internally_reserved":0,"allocated":837865472,"data_stored":837084984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":428143772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,12,13,17,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":90,"apply_latency_ms":90,"commit_latency_ns":90000000,"apply_latency_ns":90000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34062336,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.f","version":"36'3335","reported_seq":"3381","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.415538+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:39.415538+0000","last_peered":"2021-01-15T18:42:39.415538+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:39.415538+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3335,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27320320,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":26680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3407","reported_seq":"6327","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.664944+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:39.664944+0000","last_peered":"2021-01-15T18:42:39.664944+0000","last_clean":"2021-01-15T18:42:39.664944+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:39.664944+0000","last_undegraded":"2021-01-15T18:42:39.664944+0000","last_fullsized":"2021-01-15T18:42:39.664944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3407,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3381","reported_seq":"3460","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.011510+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:38.011510+0000","last_peered":"2021-01-15T18:42:38.011510+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:38.011510+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3381,"ondisk_log_size":3381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27668563,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":27026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3521","reported_seq":"3567","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.286719+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:39.286719+0000","last_peered":"2021-01-15T18:42:39.286719+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:39.286719+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3521,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28844032,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3521,"num_write_kb":28168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2089}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3543","reported_seq":"4130","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.625309+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:39.625309+0000","last_peered":"2021-01-15T18:42:39.625309+0000","last_clean":"2021-01-15T18:42:39.625309+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:39.625309+0000","last_undegraded":"2021-01-15T18:42:39.625309+0000","last_fullsized":"2021-01-15T18:42:39.625309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3543,"ondisk_log_size":3543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28975270,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":28308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3429","reported_seq":"3636","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.680620+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:39.680620+0000","last_peered":"2021-01-15T18:42:39.680620+0000","last_clean":"2021-01-15T18:42:39.680620+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:39.680620+0000","last_undegraded":"2021-01-15T18:42:39.680620+0000","last_fullsized":"2021-01-15T18:42:39.680620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3429,"ondisk_log_size":3429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28057766,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3436","reported_seq":"3490","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.280894+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:39.280894+0000","last_peered":"2021-01-15T18:42:39.280894+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:39.280894+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3436,"ondisk_log_size":3436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28106918,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":27460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3483","reported_seq":"3557","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.228892+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:38.228892+0000","last_peered":"2021-01-15T18:42:38.228892+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:38.228892+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3483,"ondisk_log_size":3483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28520531,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2030}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3520","reported_seq":"3562","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.381492+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:39.381492+0000","last_peered":"2021-01-15T18:42:39.381492+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:39.381492+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3520,"ondisk_log_size":3520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28823635,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":28149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3508","reported_seq":"6445","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.424311+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:39.424311+0000","last_peered":"2021-01-15T18:42:39.424311+0000","last_clean":"2021-01-15T18:42:39.424311+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:39.424311+0000","last_undegraded":"2021-01-15T18:42:39.424311+0000","last_fullsized":"2021-01-15T18:42:39.424311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3508,"ondisk_log_size":3508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28684537,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":28020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3412","reported_seq":"3470","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.390337+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:39.390337+0000","last_peered":"2021-01-15T18:42:39.390337+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:39.390337+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3412,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918502,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3407","reported_seq":"3470","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.033627+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:39.033627+0000","last_peered":"2021-01-15T18:42:39.033627+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:39.033627+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3407,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3404","reported_seq":"4917","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.061024+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:39.061024+0000","last_peered":"2021-01-15T18:42:39.061024+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:39.061024+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3404,"ondisk_log_size":3404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27828390,"num_objects":3398,"num_object_clones":0,"num_object_copies":13592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":27188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":725,"num_bytes_recovered":5922816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3606","reported_seq":"3670","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.112402+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:39.112402+0000","last_peered":"2021-01-15T18:42:39.112402+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:39.112402+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3606,"ondisk_log_size":3606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29491366,"num_objects":3601,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3601","reported_seq":"3659","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.393661+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:39.393661+0000","last_peered":"2021-01-15T18:42:39.393661+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:39.393661+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3601,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29417804,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":28747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3387","reported_seq":"3468","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.229800+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:38.229800+0000","last_peered":"2021-01-15T18:42:38.229800+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:38.229800+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3387,"ondisk_log_size":3387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27721894,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":27074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":453118228,"num_objects":55326,"num_object_clones":0,"num_object_copies":221304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55353,"num_write_kb":442614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5163,"num_bytes_recovered":42016768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":839483392,"data_stored":839483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55380,"ondisk_log_size":55380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330748,"kb_used_data":221564,"kb_used_omap":0,"kb_used_meta":109183,"kb_avail":94041092,"statfs":{"total":96636764160,"available":96298078208,"internally_reserved":0,"allocated":226881536,"data_stored":226686414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":111803444},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.3580000000000001},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.452}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":3.5539999999999998},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.88300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":3.8740000000000001},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.0579999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327908,"kb_used_data":219484,"kb_used_omap":0,"kb_used_meta":108416,"kb_avail":94043932,"statfs":{"total":96636764160,"available":96300986368,"internally_reserved":0,"allocated":224751616,"data_stored":224556494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":111017984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.73899999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.841}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.9359999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":1.655}]}]},{"osd":0,"up_from":22,"seq":94489280552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324864,"kb_used_data":220280,"kb_used_omap":0,"kb_used_meta":104575,"kb_avail":94046976,"statfs":{"total":96636764160,"available":96304103424,"internally_reserved":0,"allocated":225566720,"data_stored":225371598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":107084852},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.55100000000000005},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":2.8090000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.63500000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.7910000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":11.858000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":1.27}]}]},{"osd":1,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252844,"kb_used_data":156900,"kb_used_omap":0,"kb_used_meta":95935,"kb_avail":94118996,"statfs":{"total":96636764160,"available":96377851904,"internally_reserved":0,"allocated":160665600,"data_stored":160470478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":98237492},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.9649999999999999},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":3.9929999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":6.4640000000000004},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":4.1139999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":4.5570000000000004},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":4.7859999999999996}]}]}]}} 2021-01-15T18:42:41.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:42.064 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 162 20 56184 56164 2.70752 2.34375 0.0628898 0.0576913 2021-01-15T18:42:42.237 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:42.238 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16242/221304 objects degraded (7.339%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":212,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55326,"data_bytes":453118228,"bytes_used":1266036736,"bytes_avail":385281019904,"bytes_total":386547056640,"degraded_objects":16242,"degraded_total":221304,"degraded_ratio":0.073392256805118752,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":191095,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":2293760,"num_keys_recovered":0,"write_bytes_sec":2837772,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:42.294 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:42.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:42.850 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:42.931 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2021-01-15T18:42:42.418553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":453118228,"num_objects":55326,"num_object_clones":0,"num_object_copies":221304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55353,"num_write_kb":442614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5163,"num_bytes_recovered":42016768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55380,"ondisk_log_size":55380,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1236364,"kb_used_data":818228,"kb_used_omap":2,"kb_used_meta":418109,"kb_avail":376250996,"statfs":{"total":386547056640,"available":385281019904,"internally_reserved":0,"allocated":837865472,"data_stored":837084984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":428143772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,12,13,17,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":90,"apply_latency_ms":90,"commit_latency_ns":90000000,"apply_latency_ns":90000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23486464,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":22936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007578"},"pg_stats":[{"pgid":"1.f","version":"36'3335","reported_seq":"3381","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.415538+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:39.415538+0000","last_peered":"2021-01-15T18:42:39.415538+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:39.415538+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3335,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27320320,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":26680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3407","reported_seq":"6327","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.664944+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:39.664944+0000","last_peered":"2021-01-15T18:42:39.664944+0000","last_clean":"2021-01-15T18:42:39.664944+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:39.664944+0000","last_undegraded":"2021-01-15T18:42:39.664944+0000","last_fullsized":"2021-01-15T18:42:39.664944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3407,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3381","reported_seq":"3460","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.011510+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:38.011510+0000","last_peered":"2021-01-15T18:42:38.011510+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:38.011510+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3381,"ondisk_log_size":3381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27668563,"num_objects":3378,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":27026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3521","reported_seq":"3567","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.286719+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:39.286719+0000","last_peered":"2021-01-15T18:42:39.286719+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:39.286719+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3521,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28844032,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3521,"num_write_kb":28168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2089}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3543","reported_seq":"4130","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.625309+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:39.625309+0000","last_peered":"2021-01-15T18:42:39.625309+0000","last_clean":"2021-01-15T18:42:39.625309+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:39.625309+0000","last_undegraded":"2021-01-15T18:42:39.625309+0000","last_fullsized":"2021-01-15T18:42:39.625309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3543,"ondisk_log_size":3543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28975270,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":28308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3429","reported_seq":"3636","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.680620+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:39.680620+0000","last_peered":"2021-01-15T18:42:39.680620+0000","last_clean":"2021-01-15T18:42:39.680620+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:39.680620+0000","last_undegraded":"2021-01-15T18:42:39.680620+0000","last_fullsized":"2021-01-15T18:42:39.680620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3429,"ondisk_log_size":3429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28057766,"num_objects":3426,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3436","reported_seq":"3490","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.280894+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:39.280894+0000","last_peered":"2021-01-15T18:42:39.280894+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:39.280894+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3436,"ondisk_log_size":3436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28106918,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":27460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3483","reported_seq":"3557","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.228892+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:38.228892+0000","last_peered":"2021-01-15T18:42:38.228892+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:38.228892+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3483,"ondisk_log_size":3483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28520531,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2030}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3520","reported_seq":"3562","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.381492+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:39.381492+0000","last_peered":"2021-01-15T18:42:39.381492+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:39.381492+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3520,"ondisk_log_size":3520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28823635,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":28149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2083}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3508","reported_seq":"6445","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:39.424311+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:39.424311+0000","last_peered":"2021-01-15T18:42:39.424311+0000","last_clean":"2021-01-15T18:42:39.424311+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:39.424311+0000","last_undegraded":"2021-01-15T18:42:39.424311+0000","last_fullsized":"2021-01-15T18:42:39.424311+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3508,"ondisk_log_size":3508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28684537,"num_objects":3503,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":28020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3412","reported_seq":"3470","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.390337+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:39.390337+0000","last_peered":"2021-01-15T18:42:39.390337+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:39.390337+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3412,"ondisk_log_size":3412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918502,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3410,"num_write_kb":27271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2028}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3407","reported_seq":"3470","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.033627+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:39.033627+0000","last_peered":"2021-01-15T18:42:39.033627+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:39.033627+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3407,"ondisk_log_size":3407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":27223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3404","reported_seq":"4917","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.061024+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:39.061024+0000","last_peered":"2021-01-15T18:42:39.061024+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:39.061024+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3404,"ondisk_log_size":3404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27828390,"num_objects":3398,"num_object_clones":0,"num_object_copies":13592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":27188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":725,"num_bytes_recovered":5922816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2716}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3606","reported_seq":"3670","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.112402+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:39.112402+0000","last_peered":"2021-01-15T18:42:39.112402+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:39.112402+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3606,"ondisk_log_size":3606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29491366,"num_objects":3601,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3604,"num_write_kb":28812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3601","reported_seq":"3659","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:39.393661+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:39.393661+0000","last_peered":"2021-01-15T18:42:39.393661+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:39.393661+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3601,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29417804,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":28747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3387","reported_seq":"3468","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:38.229800+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:38.229800+0000","last_peered":"2021-01-15T18:42:38.229800+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:38.229800+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3387,"ondisk_log_size":3387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27721894,"num_objects":3385,"num_object_clones":0,"num_object_copies":13540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":27074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":453118228,"num_objects":55326,"num_object_clones":0,"num_object_copies":221304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55353,"num_write_kb":442614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5163,"num_bytes_recovered":42016768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":839483392,"data_stored":839483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55380,"ondisk_log_size":55380,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215142,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330748,"kb_used_data":221564,"kb_used_omap":0,"kb_used_meta":109183,"kb_avail":94041092,"statfs":{"total":96636764160,"available":96298078208,"internally_reserved":0,"allocated":226881536,"data_stored":226686414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":111803444},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.3580000000000001},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.452}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":3.5539999999999998},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.88300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":3.8740000000000001},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.0579999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327908,"kb_used_data":219484,"kb_used_omap":0,"kb_used_meta":108416,"kb_avail":94043932,"statfs":{"total":96636764160,"available":96300986368,"internally_reserved":0,"allocated":224751616,"data_stored":224556494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":111017984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.73899999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.841}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.9359999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":1.655}]}]},{"osd":0,"up_from":22,"seq":94489280552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324864,"kb_used_data":220280,"kb_used_omap":0,"kb_used_meta":104575,"kb_avail":94046976,"statfs":{"total":96636764160,"available":96304103424,"internally_reserved":0,"allocated":225566720,"data_stored":225371598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":107084852},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,4,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.55100000000000005},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":2.8090000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":0.63500000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":6.7910000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":11.858000000000001},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":1.27}]}]},{"osd":1,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252844,"kb_used_data":156900,"kb_used_omap":0,"kb_used_meta":95935,"kb_avail":94118996,"statfs":{"total":96636764160,"available":96377851904,"internally_reserved":0,"allocated":160665600,"data_stored":160470478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":98237492},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.9649999999999999},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":3.9929999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":6.4640000000000004},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":4.1139999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":4.5570000000000004},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":4.7859999999999996}]}]}]}} 2021-01-15T18:42:43.065 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 163 20 56420 56400 2.70222 1.84375 0.0638543 0.0578053 2021-01-15T18:42:44.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 164 20 56743 56723 2.70112 2.52344 0.0402451 0.0578348 2021-01-15T18:42:45.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 165 20 57048 57028 2.69919 2.38281 0.0479293 0.0578728 2021-01-15T18:42:45.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:46.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 166 20 57351 57331 2.69719 2.36719 0.0692505 0.057914 2021-01-15T18:42:46.599 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:46.668 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2021-01-15T18:42:46.419438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465209620,"num_objects":56802,"num_object_clones":0,"num_object_copies":227208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56829,"num_write_kb":454422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5251,"num_bytes_recovered":42737664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56856,"ondisk_log_size":56856,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1287708,"kb_used_data":842440,"kb_used_omap":2,"kb_used_meta":445245,"kb_avail":376199652,"statfs":{"total":386547056640,"available":385228443648,"internally_reserved":0,"allocated":862658560,"data_stored":861878072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":455931036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,4,22,17,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":87,"apply_latency_ms":87,"commit_latency_ns":87000000,"apply_latency_ns":87000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31522816,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007413"},"pg_stats":[{"pgid":"1.f","version":"36'3438","reported_seq":"3484","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.415232+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:44.415232+0000","last_peered":"2021-01-15T18:42:44.415232+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:44.415232+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3438,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3497","reported_seq":"6417","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.589993+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:44.589993+0000","last_peered":"2021-01-15T18:42:44.589993+0000","last_clean":"2021-01-15T18:42:44.589993+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:44.589993+0000","last_undegraded":"2021-01-15T18:42:44.589993+0000","last_fullsized":"2021-01-15T18:42:44.589993+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3497,"ondisk_log_size":3497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28606630,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3478","reported_seq":"3557","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:43.252054+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:43.252054+0000","last_peered":"2021-01-15T18:42:43.252054+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:43.252054+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3478,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28463187,"num_objects":3475,"num_object_clones":0,"num_object_copies":13900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3613","reported_seq":"3658","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.361747+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:44.361747+0000","last_peered":"2021-01-15T18:42:44.361747+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:44.361747+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3613,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3636","reported_seq":"4224","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.616530+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:44.616530+0000","last_peered":"2021-01-15T18:42:44.616530+0000","last_clean":"2021-01-15T18:42:44.616530+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:44.616530+0000","last_undegraded":"2021-01-15T18:42:44.616530+0000","last_fullsized":"2021-01-15T18:42:44.616530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3636,"ondisk_log_size":3636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29737126,"num_objects":3631,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":29052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3513","reported_seq":"3720","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.408807+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:44.408807+0000","last_peered":"2021-01-15T18:42:44.408807+0000","last_clean":"2021-01-15T18:42:44.408807+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:44.408807+0000","last_undegraded":"2021-01-15T18:42:44.408807+0000","last_fullsized":"2021-01-15T18:42:44.408807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3513,"ondisk_log_size":3513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28745894,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3511,"num_write_kb":28079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3522","reported_seq":"3576","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.384506+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:44.384506+0000","last_peered":"2021-01-15T18:42:44.384506+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:44.384506+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3522,"ondisk_log_size":3522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28811430,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3520,"num_write_kb":28148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3587","reported_seq":"3660","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:43.150927+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:43.150927+0000","last_peered":"2021-01-15T18:42:43.150927+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:43.150927+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3587,"ondisk_log_size":3587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29372499,"num_objects":3586,"num_object_clones":0,"num_object_copies":14344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3586,"num_write_kb":28685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3613","reported_seq":"3655","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.275314+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:44.275314+0000","last_peered":"2021-01-15T18:42:44.275314+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:44.275314+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3613,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29585491,"num_objects":3612,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":28893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3600","reported_seq":"6537","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.383043+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:44.383043+0000","last_peered":"2021-01-15T18:42:44.383043+0000","last_clean":"2021-01-15T18:42:44.383043+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:44.383043+0000","last_undegraded":"2021-01-15T18:42:44.383043+0000","last_fullsized":"2021-01-15T18:42:44.383043+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3600,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29438201,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3491","reported_seq":"3548","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.399875+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:44.399875+0000","last_peered":"2021-01-15T18:42:44.399875+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:44.399875+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3491,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28565670,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3484","reported_seq":"3548","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.109362+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:44.109362+0000","last_peered":"2021-01-15T18:42:44.109362+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:44.109362+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3484,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28500134,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3500","reported_seq":"5189","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.097319+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:44.097319+0000","last_peered":"2021-01-15T18:42:44.097319+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:44.097319+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3500,"ondisk_log_size":3500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28614822,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":813,"num_bytes_recovered":6643712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2900}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3705","reported_seq":"3768","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:43.975168+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:43.975168+0000","last_peered":"2021-01-15T18:42:43.975168+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:43.975168+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3705,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30302374,"num_objects":3700,"num_object_clones":0,"num_object_copies":14800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3695","reported_seq":"3755","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.321566+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:44.321566+0000","last_peered":"2021-01-15T18:42:44.321566+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:44.321566+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3695,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30187852,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":29499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3484","reported_seq":"3565","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:43.122679+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:43.122679+0000","last_peered":"2021-01-15T18:42:43.122679+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:43.122679+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3484,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28516518,"num_objects":3482,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2100}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465209620,"num_objects":56802,"num_object_clones":0,"num_object_copies":227208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56829,"num_write_kb":454422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5251,"num_bytes_recovered":42737664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":863911936,"data_stored":863911936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56856,"ondisk_log_size":56856,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347648,"kb_used_data":227576,"kb_used_omap":0,"kb_used_meta":120063,"kb_avail":94024192,"statfs":{"total":96636764160,"available":96280772608,"internally_reserved":0,"allocated":233037824,"data_stored":232842702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":122944564},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,14,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":13.763},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":13.858000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":12.227},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":10.095000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":15.164},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":13.662000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344844,"kb_used_data":225540,"kb_used_omap":0,"kb_used_meta":119296,"kb_avail":94026996,"statfs":{"total":96636764160,"available":96283643904,"internally_reserved":0,"allocated":230952960,"data_stored":230757838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":122159104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:48 2021","interfaces":[{"interface":"back","average":{"1min":4.4560000000000004,"5min":3.895,"15min":3.8010000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":22.417999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.6739999999999999},{"interface":"front","average":{"1min":3.0790000000000002,"5min":1.4039999999999999,"15min":1.125},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":21.344999999999999,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.626}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":8.3239999999999998},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":6.9610000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":2.266},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":2.9399999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341616,"kb_used_data":226220,"kb_used_omap":0,"kb_used_meta":115391,"kb_avail":94030224,"statfs":{"total":96636764160,"available":96286949376,"internally_reserved":0,"allocated":231649280,"data_stored":231454158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":118160436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.65000000000000002},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":2.5670000000000002},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.2880000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":16.978999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":17.088999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":253600,"kb_used_data":163104,"kb_used_omap":0,"kb_used_meta":90495,"kb_avail":94118240,"statfs":{"total":96636764160,"available":96377077760,"internally_reserved":0,"allocated":167018496,"data_stored":166823374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92666932},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.609},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.873}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":4.548},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":4.7030000000000003}]}]}]}} 2021-01-15T18:42:46.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:47.068 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 167 20 57604 57584 2.69287 1.97656 0.0573793 0.0580015 2021-01-15T18:42:47.984 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:47.985 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16154/227208 objects degraded (7.110%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":218,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56802,"data_bytes":465209620,"bytes_used":1318612992,"bytes_avail":385228443648,"bytes_total":386547056640,"degraded_objects":16154,"degraded_total":227208,"degraded_ratio":0.071097848667300442,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":118710,"recovering_keys_per_sec":0,"num_objects_recovered":174,"num_bytes_recovered":1425408,"num_keys_recovered":0,"write_bytes_sec":2625279,"read_op_per_sec":0,"write_op_per_sec":320},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:48.055 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:48.056 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:48.072 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 168 20 57899 57879 2.69056 2.30469 0.0909819 0.0580549 2021-01-15T18:42:49.068 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 169 20 58202 58182 2.68865 2.36719 0.0370514 0.0581053 2021-01-15T18:42:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:49.320 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2021-01-15T18:42:48.419996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":467478804,"num_objects":57079,"num_object_clones":0,"num_object_copies":228316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57106,"num_write_kb":456638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5251,"num_bytes_recovered":42737664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57133,"ondisk_log_size":57133,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1288292,"kb_used_data":848528,"kb_used_omap":2,"kb_used_meta":439741,"kb_avail":376199068,"statfs":{"total":386547056640,"available":385227845632,"internally_reserved":0,"allocated":868892672,"data_stored":868112184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":450294940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,23,12,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":88,"apply_latency_ms":88,"commit_latency_ns":88000000,"apply_latency_ns":88000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27074560,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":26440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007509"},"pg_stats":[{"pgid":"1.f","version":"36'3438","reported_seq":"3484","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.415232+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:44.415232+0000","last_peered":"2021-01-15T18:42:44.415232+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:44.415232+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3438,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28164096,"num_objects":3438,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3438,"num_write_kb":27504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2084}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3497","reported_seq":"6417","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.589993+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:44.589993+0000","last_peered":"2021-01-15T18:42:44.589993+0000","last_clean":"2021-01-15T18:42:44.589993+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:44.589993+0000","last_undegraded":"2021-01-15T18:42:44.589993+0000","last_fullsized":"2021-01-15T18:42:44.589993+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3497,"ondisk_log_size":3497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28606630,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3574","reported_seq":"3653","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.199769+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:48.199769+0000","last_peered":"2021-01-15T18:42:48.199769+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:48.199769+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3574,"ondisk_log_size":3574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29249619,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":28570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3613","reported_seq":"3658","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.361747+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:44.361747+0000","last_peered":"2021-01-15T18:42:44.361747+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:44.361747+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3613,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29597696,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":28904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3636","reported_seq":"4224","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.616530+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:44.616530+0000","last_peered":"2021-01-15T18:42:44.616530+0000","last_clean":"2021-01-15T18:42:44.616530+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:44.616530+0000","last_undegraded":"2021-01-15T18:42:44.616530+0000","last_fullsized":"2021-01-15T18:42:44.616530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3636,"ondisk_log_size":3636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29737126,"num_objects":3631,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":29052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3513","reported_seq":"3720","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.408807+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:44.408807+0000","last_peered":"2021-01-15T18:42:44.408807+0000","last_clean":"2021-01-15T18:42:44.408807+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:44.408807+0000","last_undegraded":"2021-01-15T18:42:44.408807+0000","last_fullsized":"2021-01-15T18:42:44.408807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3513,"ondisk_log_size":3513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28745894,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3511,"num_write_kb":28079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3522","reported_seq":"3576","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.384506+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:44.384506+0000","last_peered":"2021-01-15T18:42:44.384506+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:44.384506+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3522,"ondisk_log_size":3522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28811430,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3520,"num_write_kb":28148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3669","reported_seq":"3740","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.259408+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:48.259408+0000","last_peered":"2021-01-15T18:42:48.259408+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:48.259408+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3669,"ondisk_log_size":3669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30044243,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3613","reported_seq":"3655","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.275314+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:44.275314+0000","last_peered":"2021-01-15T18:42:44.275314+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:44.275314+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3613,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29585491,"num_objects":3612,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":28893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2176}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3600","reported_seq":"6537","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:44.383043+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:44.383043+0000","last_peered":"2021-01-15T18:42:44.383043+0000","last_clean":"2021-01-15T18:42:44.383043+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:44.383043+0000","last_undegraded":"2021-01-15T18:42:44.383043+0000","last_fullsized":"2021-01-15T18:42:44.383043+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3600,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29438201,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3491","reported_seq":"3548","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.399875+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:44.399875+0000","last_peered":"2021-01-15T18:42:44.399875+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:44.399875+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3491,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28565670,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3484","reported_seq":"3548","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.109362+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:44.109362+0000","last_peered":"2021-01-15T18:42:44.109362+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:44.109362+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3484,"ondisk_log_size":3484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28500134,"num_objects":3480,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":27839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3500","reported_seq":"5189","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.097319+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:44.097319+0000","last_peered":"2021-01-15T18:42:44.097319+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:44.097319+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3500,"ondisk_log_size":3500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28614822,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":27956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":813,"num_bytes_recovered":6643712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2900}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3705","reported_seq":"3768","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:43.975168+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:43.975168+0000","last_peered":"2021-01-15T18:42:43.975168+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:43.975168+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3705,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30302374,"num_objects":3700,"num_object_clones":0,"num_object_copies":14800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3695","reported_seq":"3755","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:44.321566+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:44.321566+0000","last_peered":"2021-01-15T18:42:44.321566+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:44.321566+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3695,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30187852,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":29499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3583","reported_seq":"3663","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.170363+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:48.170363+0000","last_peered":"2021-01-15T18:42:48.170363+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:48.170363+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3583,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29327526,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3581,"num_write_kb":28642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":467478804,"num_objects":57079,"num_object_clones":0,"num_object_copies":228316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57106,"num_write_kb":456638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5251,"num_bytes_recovered":42737664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":869957632,"data_stored":869957632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57133,"ondisk_log_size":57133,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347648,"kb_used_data":227576,"kb_used_omap":0,"kb_used_meta":120063,"kb_avail":94024192,"statfs":{"total":96636764160,"available":96280772608,"internally_reserved":0,"allocated":233037824,"data_stored":232842702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":122944564},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,14,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":13.763},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":13.858000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":12.227},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":10.095000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":15.164},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":13.662000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345428,"kb_used_data":231628,"kb_used_omap":0,"kb_used_meta":113792,"kb_avail":94026412,"statfs":{"total":96636764160,"available":96283045888,"internally_reserved":0,"allocated":237187072,"data_stored":236991950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":116523008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":2.375},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":3.1579999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.8049999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":0.75600000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341616,"kb_used_data":226220,"kb_used_omap":0,"kb_used_meta":115391,"kb_avail":94030224,"statfs":{"total":96636764160,"available":96286949376,"internally_reserved":0,"allocated":231649280,"data_stored":231454158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":118160436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.65000000000000002},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:41:47 2021","interfaces":[{"interface":"back","average":{"1min":4.1849999999999996,"5min":2.613,"15min":2.351},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":27.77,"5min":27.77,"15min":27.77},"last":2.5670000000000002},{"interface":"front","average":{"1min":4.54,"5min":2.7650000000000001,"15min":2.4689999999999999},"min":{"1min":0.51200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":22.765000000000001,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.2880000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":16.978999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":17.088999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":253600,"kb_used_data":163104,"kb_used_omap":0,"kb_used_meta":90495,"kb_avail":94118240,"statfs":{"total":96636764160,"available":96377077760,"internally_reserved":0,"allocated":167018496,"data_stored":166823374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":92666932},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.609},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.873}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":4.548},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":4.7030000000000003}]}]}]}} 2021-01-15T18:42:50.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 170 20 58493 58473 2.6862 2.27344 0.0250157 0.058135 2021-01-15T18:42:51.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 171 20 58743 58723 2.68192 1.95312 0.0635573 0.0582447 2021-01-15T18:42:52.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 172 20 59033 59013 2.67949 2.26562 0.0419545 0.0582963 2021-01-15T18:42:52.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:53.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 173 20 59339 59319 2.67782 2.39062 0.0747841 0.0583361 2021-01-15T18:42:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:53.520 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:53.542 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2021-01-15T18:42:52.423069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477137172,"num_objects":58258,"num_object_clones":0,"num_object_copies":233032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58285,"num_write_kb":466070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5377,"num_bytes_recovered":43753472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58312,"ondisk_log_size":58312,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1318160,"kb_used_data":866372,"kb_used_omap":2,"kb_used_meta":451773,"kb_avail":376169200,"statfs":{"total":386547056640,"available":385197260800,"internally_reserved":0,"allocated":887164928,"data_stored":886384440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":462615708},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,9,18,16,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":90,"apply_latency_ms":90,"commit_latency_ns":90000000,"apply_latency_ns":90000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24018944,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":23456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009427"},"pg_stats":[{"pgid":"1.f","version":"36'3535","reported_seq":"3580","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.429184+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:49.429184+0000","last_peered":"2021-01-15T18:42:49.429184+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:49.429184+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3535,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958720,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3535,"num_write_kb":28280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3577","reported_seq":"6496","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.691503+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:49.691503+0000","last_peered":"2021-01-15T18:42:49.691503+0000","last_clean":"2021-01-15T18:42:49.691503+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:49.691503+0000","last_undegraded":"2021-01-15T18:42:49.691503+0000","last_fullsized":"2021-01-15T18:42:49.691503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3577,"ondisk_log_size":3577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29261990,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3574","reported_seq":"3653","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.199769+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:48.199769+0000","last_peered":"2021-01-15T18:42:48.199769+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:48.199769+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3574,"ondisk_log_size":3574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29249619,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":28570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3711","reported_seq":"3757","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.258909+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:49.258909+0000","last_peered":"2021-01-15T18:42:49.258909+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:49.258909+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3711,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30400512,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3711,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3716","reported_seq":"4304","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.454960+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:49.454960+0000","last_peered":"2021-01-15T18:42:49.454960+0000","last_clean":"2021-01-15T18:42:49.454960+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:49.454960+0000","last_undegraded":"2021-01-15T18:42:49.454960+0000","last_fullsized":"2021-01-15T18:42:49.454960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3716,"ondisk_log_size":3716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30392486,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3714,"num_write_kb":29692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3599","reported_seq":"3806","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.688928+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:49.688928+0000","last_peered":"2021-01-15T18:42:49.688928+0000","last_clean":"2021-01-15T18:42:49.688928+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:49.688928+0000","last_undegraded":"2021-01-15T18:42:49.688928+0000","last_fullsized":"2021-01-15T18:42:49.688928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3599,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29450406,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3628","reported_seq":"3681","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.394720+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:49.394720+0000","last_peered":"2021-01-15T18:42:49.394720+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:49.394720+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3628,"ondisk_log_size":3628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29679782,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3669","reported_seq":"3740","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.259408+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:48.259408+0000","last_peered":"2021-01-15T18:42:48.259408+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:48.259408+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3669,"ondisk_log_size":3669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30044243,"num_objects":3668,"num_object_clones":0,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3668,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3705","reported_seq":"3744","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.485413+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:49.485413+0000","last_peered":"2021-01-15T18:42:49.485413+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:49.485413+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3705,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30339155,"num_objects":3704,"num_object_clones":0,"num_object_copies":14816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":29629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2268}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3688","reported_seq":"6624","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.409272+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:49.409272+0000","last_peered":"2021-01-15T18:42:49.409272+0000","last_clean":"2021-01-15T18:42:49.409272+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:49.409272+0000","last_undegraded":"2021-01-15T18:42:49.409272+0000","last_fullsized":"2021-01-15T18:42:49.409272+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3688,"ondisk_log_size":3688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30159097,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":29460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3584","reported_seq":"3642","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.413223+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:49.413223+0000","last_peered":"2021-01-15T18:42:49.413223+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:49.413223+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3584,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29327526,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3570","reported_seq":"3634","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.050560+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:49.050560+0000","last_peered":"2021-01-15T18:42:49.050560+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:49.050560+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3570,"ondisk_log_size":3570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29204646,"num_objects":3566,"num_object_clones":0,"num_object_copies":14264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":28527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3589","reported_seq":"5528","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.129748+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:49.129748+0000","last_peered":"2021-01-15T18:42:49.129748+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:49.129748+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3589,"ondisk_log_size":3589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29343910,"num_objects":3583,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":7659520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3790","reported_seq":"3855","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.048859+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:49.048859+0000","last_peered":"2021-01-15T18:42:49.048859+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:49.048859+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3790,"ondisk_log_size":3790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30998694,"num_objects":3785,"num_object_clones":0,"num_object_copies":15140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":30284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3794","reported_seq":"3850","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.475913+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:49.475913+0000","last_peered":"2021-01-15T18:42:49.475913+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:49.475913+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3794,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30998860,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3583","reported_seq":"3663","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:48.170363+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:48.170363+0000","last_peered":"2021-01-15T18:42:48.170363+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:48.170363+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3583,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29327526,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3581,"num_write_kb":28642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":477137172,"num_objects":58258,"num_object_clones":0,"num_object_copies":233032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58285,"num_write_kb":466070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5377,"num_bytes_recovered":43753472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":888438784,"data_stored":888438784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58312,"ondisk_log_size":58312,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343560,"kb_used_data":233280,"kb_used_omap":0,"kb_used_meta":110271,"kb_avail":94028280,"statfs":{"total":96636764160,"available":96284958720,"internally_reserved":0,"allocated":238878720,"data_stored":238683598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":112917556},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,7,8,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.286},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.3730000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":2.714},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":3.9049999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":2.673},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":3.7170000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345428,"kb_used_data":231628,"kb_used_omap":0,"kb_used_meta":113792,"kb_avail":94026412,"statfs":{"total":96636764160,"available":96283045888,"internally_reserved":0,"allocated":237187072,"data_stored":236991950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":116523008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":2.375},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":3.1579999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":2.165,"15min":2.2480000000000002},"min":{"1min":0.49299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.3719999999999999,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.8049999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.2480000000000002,"15min":2.2639999999999998},"min":{"1min":0.53300000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.9740000000000002,"5min":9.9740000000000002,"15min":9.9740000000000002},"last":0.75600000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280554,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358328,"kb_used_data":232112,"kb_used_omap":0,"kb_used_meta":126207,"kb_avail":94013512,"statfs":{"total":96636764160,"available":96269836288,"internally_reserved":0,"allocated":237682688,"data_stored":237487566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":129236020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":8.6590000000000007},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":3.359}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":8.8390000000000004},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.9370000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":0.71699999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270844,"kb_used_data":169352,"kb_used_omap":0,"kb_used_meta":101503,"kb_avail":94100996,"statfs":{"total":96636764160,"available":96359419904,"internally_reserved":0,"allocated":173416448,"data_stored":173221326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":103939124},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":7.1449999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.753},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.3340000000000001}]}]}]}} 2021-01-15T18:42:53.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:42:54.066 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 174 20 59626 59606 2.67532 2.24219 0.0441871 0.0583759 2021-01-15T18:42:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:54.802 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15905/235292 objects degraded (6.760%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":225,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58823,"data_bytes":481765652,"bytes_used":1362636800,"bytes_avail":385184419840,"bytes_total":386547056640,"degraded_objects":15905,"degraded_total":235292,"degraded_ratio":0.067596858371725341,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":229929,"recovering_keys_per_sec":0,"num_objects_recovered":339,"num_bytes_recovered":2760704,"num_keys_recovered":0,"write_bytes_sec":2385948,"read_op_per_sec":0,"write_op_per_sec":291},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:42:54.873 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:42:54.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:42:55.069 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 175 20 59899 59879 2.67218 2.13281 0.0942994 0.0584543 2021-01-15T18:42:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:42:56.024 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:42:56.069 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 176 20 60196 60176 2.67018 2.32031 0.0609094 0.0585038 2021-01-15T18:42:56.083 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2021-01-15T18:42:54.425277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481765652,"num_objects":58823,"num_object_clones":0,"num_object_copies":235292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58850,"num_write_kb":470590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5502,"num_bytes_recovered":44777472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58877,"ondisk_log_size":58877,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1330700,"kb_used_data":877940,"kb_used_omap":2,"kb_used_meta":452797,"kb_avail":376156660,"statfs":{"total":386547056640,"available":385184419840,"internally_reserved":0,"allocated":899010560,"data_stored":898230072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":463664284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,12,17,21,24,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28647424,"num_objects":3497,"num_object_clones":0,"num_object_copies":13988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3497,"num_write_kb":27976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":2760704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006723"},"pg_stats":[{"pgid":"1.f","version":"36'3535","reported_seq":"3580","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.429184+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:49.429184+0000","last_peered":"2021-01-15T18:42:49.429184+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:49.429184+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3535,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958720,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3535,"num_write_kb":28280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2181}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3577","reported_seq":"6496","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.691503+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:49.691503+0000","last_peered":"2021-01-15T18:42:49.691503+0000","last_clean":"2021-01-15T18:42:49.691503+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:49.691503+0000","last_undegraded":"2021-01-15T18:42:49.691503+0000","last_fullsized":"2021-01-15T18:42:49.691503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3577,"ondisk_log_size":3577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29261990,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":28588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3676","reported_seq":"3755","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:53.232547+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:53.232547+0000","last_peered":"2021-01-15T18:42:53.232547+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:53.232547+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3676,"ondisk_log_size":3676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30085203,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3675,"num_write_kb":29386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3711","reported_seq":"3757","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.258909+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:49.258909+0000","last_peered":"2021-01-15T18:42:49.258909+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:49.258909+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3711,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30400512,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3711,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3716","reported_seq":"4304","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.454960+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:49.454960+0000","last_peered":"2021-01-15T18:42:49.454960+0000","last_clean":"2021-01-15T18:42:49.454960+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:49.454960+0000","last_undegraded":"2021-01-15T18:42:49.454960+0000","last_fullsized":"2021-01-15T18:42:49.454960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3716,"ondisk_log_size":3716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30392486,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3714,"num_write_kb":29692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3599","reported_seq":"3806","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.688928+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:49.688928+0000","last_peered":"2021-01-15T18:42:49.688928+0000","last_clean":"2021-01-15T18:42:49.688928+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:49.688928+0000","last_undegraded":"2021-01-15T18:42:49.688928+0000","last_fullsized":"2021-01-15T18:42:49.688928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3599,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29450406,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3628","reported_seq":"3681","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.394720+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:49.394720+0000","last_peered":"2021-01-15T18:42:49.394720+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:49.394720+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3628,"ondisk_log_size":3628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29679782,"num_objects":3624,"num_object_clones":0,"num_object_copies":14496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3626,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2232}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3777","reported_seq":"3851","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:53.228001+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:53.228001+0000","last_peered":"2021-01-15T18:42:53.228001+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:53.228001+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3777,"ondisk_log_size":3777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30928979,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":30205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3705","reported_seq":"3744","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.485413+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:49.485413+0000","last_peered":"2021-01-15T18:42:49.485413+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:49.485413+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3705,"ondisk_log_size":3705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30339155,"num_objects":3704,"num_object_clones":0,"num_object_copies":14816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":29629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2268}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3688","reported_seq":"6624","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:49.409272+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:49.409272+0000","last_peered":"2021-01-15T18:42:49.409272+0000","last_clean":"2021-01-15T18:42:49.409272+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:49.409272+0000","last_undegraded":"2021-01-15T18:42:49.409272+0000","last_fullsized":"2021-01-15T18:42:49.409272+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3688,"ondisk_log_size":3688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30159097,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":29460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3584","reported_seq":"3642","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.413223+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:49.413223+0000","last_peered":"2021-01-15T18:42:49.413223+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:49.413223+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3584,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29327526,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3657","reported_seq":"3721","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:53.772700+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:53.772700+0000","last_peered":"2021-01-15T18:42:53.772700+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:53.772700+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3657,"ondisk_log_size":3657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29917350,"num_objects":3653,"num_object_clones":0,"num_object_copies":14612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":29223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3686","reported_seq":"5874","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.102508+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:54.102508+0000","last_peered":"2021-01-15T18:42:54.102508+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:54.102508+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3686,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30138534,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1064,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3862","reported_seq":"3926","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.136763+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:54.136763+0000","last_peered":"2021-01-15T18:42:54.136763+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:54.136763+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3862,"ondisk_log_size":3862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31588518,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":30860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3794","reported_seq":"3850","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:49.475913+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:49.475913+0000","last_peered":"2021-01-15T18:42:49.475913+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:49.475913+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3794,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30998860,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3682","reported_seq":"3763","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:53.177308+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:53.177308+0000","last_peered":"2021-01-15T18:42:53.177308+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:53.177308+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3682,"ondisk_log_size":3682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30138534,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3680,"num_write_kb":29434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2298}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":481765652,"num_objects":58823,"num_object_clones":0,"num_object_copies":235292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58850,"num_write_kb":470590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5502,"num_bytes_recovered":44777472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":900165632,"data_stored":900165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58877,"ondisk_log_size":58877,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215144,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343560,"kb_used_data":233280,"kb_used_omap":0,"kb_used_meta":110271,"kb_avail":94028280,"statfs":{"total":96636764160,"available":96284958720,"internally_reserved":0,"allocated":238878720,"data_stored":238683598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":112917556},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,7,8,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.286},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":3.3730000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":2.714},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":3.9049999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":2.673},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":3.7170000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361792,"kb_used_data":237356,"kb_used_omap":0,"kb_used_meta":124480,"kb_avail":94010048,"statfs":{"total":96636764160,"available":96266289152,"internally_reserved":0,"allocated":243052544,"data_stored":242857422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":127467520},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":4.0010000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":4.125},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":1.0680000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354504,"kb_used_data":237952,"kb_used_omap":0,"kb_used_meta":116543,"kb_avail":94017336,"statfs":{"total":96636764160,"available":96273752064,"internally_reserved":0,"allocated":243662848,"data_stored":243467726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":119340084},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.89600000000000002},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":2.9649999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.58899999999999997},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":5.9969999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.8959999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":3.488}]}]},{"osd":1,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":270844,"kb_used_data":169352,"kb_used_omap":0,"kb_used_meta":101503,"kb_avail":94100996,"statfs":{"total":96636764160,"available":96359419904,"internally_reserved":0,"allocated":173416448,"data_stored":173221326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":103939124},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":7.1449999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.753},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.3340000000000001}]}]}]}} 2021-01-15T18:42:57.070 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 177 20 60493 60473 2.66819 2.32031 0.0558996 0.0585384 2021-01-15T18:42:58.070 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 178 20 60802 60782 2.66677 2.41406 0.0723444 0.0585754 2021-01-15T18:42:59.070 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 179 20 61075 61055 2.66378 2.13281 0.0682143 0.0586452 2021-01-15T18:42:59.086 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:00.073 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:43:00.070955+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.058694 2021-01-15T18:43:00.073 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:43:00.073 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 180 20 61362 61342 2.66143 2.24219 0.147318 0.058694 2021-01-15T18:43:00.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:00.330 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:00.354 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2021-01-15T18:42:58.430418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491088148,"num_objects":59961,"num_object_clones":0,"num_object_copies":239844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59988,"num_write_kb":479694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5502,"num_bytes_recovered":44777472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60015,"ondisk_log_size":60015,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1364588,"kb_used_data":895736,"kb_used_omap":2,"kb_used_meta":468925,"kb_avail":376122772,"statfs":{"total":386547056640,"available":385149718528,"internally_reserved":0,"allocated":917233664,"data_stored":916453176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":480179356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,10,13,29,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25878528,"num_objects":3159,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3159,"num_write_kb":25272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2039808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010979"},"pg_stats":[{"pgid":"1.f","version":"36'3614","reported_seq":"3660","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.449249+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:54.449249+0000","last_peered":"2021-01-15T18:42:54.449249+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:54.449249+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3614,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3658","reported_seq":"6578","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:54.433605+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:54.433605+0000","last_peered":"2021-01-15T18:42:54.433605+0000","last_clean":"2021-01-15T18:42:54.433605+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:54.433605+0000","last_undegraded":"2021-01-15T18:42:54.433605+0000","last_fullsized":"2021-01-15T18:42:54.433605+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3658,"ondisk_log_size":3658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29925542,"num_objects":3654,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":29236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3748","reported_seq":"3827","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.134829+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:58.134829+0000","last_peered":"2021-01-15T18:42:58.134829+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:58.134829+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3748,"ondisk_log_size":3748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30675027,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3747,"num_write_kb":29962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2329}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3795","reported_seq":"3841","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.461531+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:54.461531+0000","last_peered":"2021-01-15T18:42:54.461531+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:54.461531+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3795,"ondisk_log_size":3795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31088640,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":30360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3805","reported_seq":"4393","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:54.530416+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:54.530416+0000","last_peered":"2021-01-15T18:42:54.530416+0000","last_clean":"2021-01-15T18:42:54.530416+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:54.530416+0000","last_undegraded":"2021-01-15T18:42:54.530416+0000","last_fullsized":"2021-01-15T18:42:54.530416+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3805,"ondisk_log_size":3805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31121574,"num_objects":3800,"num_object_clones":0,"num_object_copies":15200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":30404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3689","reported_seq":"3896","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:54.493821+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:54.493821+0000","last_peered":"2021-01-15T18:42:54.493821+0000","last_clean":"2021-01-15T18:42:54.493821+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:54.493821+0000","last_undegraded":"2021-01-15T18:42:54.493821+0000","last_fullsized":"2021-01-15T18:42:54.493821+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3689,"ondisk_log_size":3689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30187686,"num_objects":3686,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":29487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3711","reported_seq":"3762","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.460774+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:54.460774+0000","last_peered":"2021-01-15T18:42:54.460774+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:54.460774+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3711,"ondisk_log_size":3711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30359718,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3709,"num_write_kb":29660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3863","reported_seq":"3937","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.208178+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:58.208178+0000","last_peered":"2021-01-15T18:42:58.208178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:58.208178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3863,"ondisk_log_size":3863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31633491,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3862,"num_write_kb":30893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3797","reported_seq":"3839","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.431095+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:54.431095+0000","last_peered":"2021-01-15T18:42:54.431095+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:54.431095+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3797,"ondisk_log_size":3797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31092819,"num_objects":3796,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":30365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2360}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3773","reported_seq":"6708","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:54.479578+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:54.479578+0000","last_peered":"2021-01-15T18:42:54.479578+0000","last_clean":"2021-01-15T18:42:54.479578+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:54.479578+0000","last_undegraded":"2021-01-15T18:42:54.479578+0000","last_fullsized":"2021-01-15T18:42:54.479578+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3773,"ondisk_log_size":3773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30855417,"num_objects":3768,"num_object_clones":0,"num_object_copies":15072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":30140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3669","reported_seq":"3727","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.284707+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:54.284707+0000","last_peered":"2021-01-15T18:42:54.284707+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:54.284707+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3669,"ondisk_log_size":3669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30023846,"num_objects":3666,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":29327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2285}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3657","reported_seq":"3721","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:53.772700+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:53.772700+0000","last_peered":"2021-01-15T18:42:53.772700+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:53.772700+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3657,"ondisk_log_size":3657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29917350,"num_objects":3653,"num_object_clones":0,"num_object_copies":14612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":29223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3686","reported_seq":"5874","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.102508+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:54.102508+0000","last_peered":"2021-01-15T18:42:54.102508+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:54.102508+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3686,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30138534,"num_objects":3680,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1064,"num_bytes_recovered":8683520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3862","reported_seq":"3926","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.136763+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:54.136763+0000","last_peered":"2021-01-15T18:42:54.136763+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:54.136763+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3862,"ondisk_log_size":3862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31588518,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":30860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3900","reported_seq":"3960","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:54.476078+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:54.476078+0000","last_peered":"2021-01-15T18:42:54.476078+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:54.476078+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3900,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31867212,"num_objects":3892,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3897,"num_write_kb":31139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2406}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3788","reported_seq":"3869","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.181480+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:58.181480+0000","last_peered":"2021-01-15T18:42:58.181480+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:58.181480+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3788,"ondisk_log_size":3788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31006886,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":491088148,"num_objects":59961,"num_object_clones":0,"num_object_copies":239844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59988,"num_write_kb":479694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5502,"num_bytes_recovered":44777472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":918294528,"data_stored":918294528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60015,"ondisk_log_size":60015,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215145,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359904,"kb_used_data":238944,"kb_used_omap":0,"kb_used_meta":120959,"kb_avail":94011936,"statfs":{"total":96636764160,"available":96268222464,"internally_reserved":0,"allocated":244678656,"data_stored":244483534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":123862068},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,11,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.0110000000000001,"15min":3.0089999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.8049999999999997},{"interface":"front","average":{"1min":1.8120000000000001,"5min":3.1360000000000001,"15min":3.3559999999999999},"min":{"1min":0.48999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.406000000000001,"5min":14.435,"15min":14.435},"last":5.0830000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":4.9199999999999999},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":4.8739999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:41:52 2021","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.302,"15min":2.097},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":12.91,"5min":12.91,"15min":12.91},"last":8.2680000000000007},{"interface":"front","average":{"1min":3.105,"5min":2.8980000000000001,"15min":2.8639999999999999},"min":{"1min":0.52500000000000002,"5min":0.503,"15min":0.503},"max":{"1min":18.300000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.8070000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362160,"kb_used_data":243272,"kb_used_omap":0,"kb_used_meta":118976,"kb_avail":94009680,"statfs":{"total":96636764160,"available":96265912320,"internally_reserved":0,"allocated":249110528,"data_stored":248915406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":121831424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.3479999999999999},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.96}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":9.0779999999999994},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":2.8340000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":2.4489999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354504,"kb_used_data":237952,"kb_used_omap":0,"kb_used_meta":116543,"kb_avail":94017336,"statfs":{"total":96636764160,"available":96273752064,"internally_reserved":0,"allocated":243662848,"data_stored":243467726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":119340084},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.89600000000000002},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":2.9649999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.58899999999999997},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":5.9969999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":4.0590000000000002,"15min":4.4820000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":7.202,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.8959999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":2.302,"15min":2.1880000000000002},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":14.670999999999999,"5min":14.670999999999999,"15min":14.670999999999999},"last":3.488}]}]},{"osd":1,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288020,"kb_used_data":175568,"kb_used_omap":0,"kb_used_meta":112447,"kb_avail":94083820,"statfs":{"total":96636764160,"available":96341831680,"internally_reserved":0,"allocated":179781632,"data_stored":179586510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":115145780},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.087},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.228}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":1.554}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.446}]}]}]}} 2021-01-15T18:43:00.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:01.070 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 181 20 61654 61634 2.65933 2.28125 0.0724903 0.0587392 2021-01-15T18:43:01.469 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:01.469 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15811/244568 objects degraded (6.465%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":231,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61142,"data_bytes":500762900,"bytes_used":1427795968,"bytes_avail":385119260672,"bytes_total":386547056640,"degraded_objects":15811,"degraded_total":244568,"degraded_ratio":0.064648686663831736,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":233891,"recovering_keys_per_sec":0,"num_objects_recovered":345,"num_bytes_recovered":2809856,"num_keys_recovered":0,"write_bytes_sec":2770551,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:01.533 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:01.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:02.071 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 182 20 61855 61835 2.65335 1.57031 0.0923587 0.0588732 2021-01-15T18:43:02.193 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:02.222 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2021-01-15T18:43:00.433523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":500762900,"num_objects":61142,"num_object_clones":0,"num_object_copies":244568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61169,"num_write_kb":489142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5596,"num_bytes_recovered":45547520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61196,"ondisk_log_size":61196,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1394332,"kb_used_data":913500,"kb_used_omap":2,"kb_used_meta":480893,"kb_avail":376093028,"statfs":{"total":386547056640,"available":385119260672,"internally_reserved":0,"allocated":935424000,"data_stored":934643512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":492434588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,6,14,9,16,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33284096,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":2809856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013526"},"pg_stats":[{"pgid":"1.f","version":"36'3711","reported_seq":"3757","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.453409+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:59.453409+0000","last_peered":"2021-01-15T18:42:59.453409+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:59.453409+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3711,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30400512,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3711,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3746","reported_seq":"6665","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.678824+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:59.678824+0000","last_peered":"2021-01-15T18:42:59.678824+0000","last_clean":"2021-01-15T18:42:59.678824+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:59.678824+0000","last_undegraded":"2021-01-15T18:42:59.678824+0000","last_fullsized":"2021-01-15T18:42:59.678824+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3746,"ondisk_log_size":3746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30646438,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3748","reported_seq":"3827","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.134829+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:42:58.134829+0000","last_peered":"2021-01-15T18:42:58.134829+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:42:58.134829+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3748,"ondisk_log_size":3748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30675027,"num_objects":3745,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3747,"num_write_kb":29962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2329}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3876","reported_seq":"3922","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.326282+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:59.326282+0000","last_peered":"2021-01-15T18:42:59.326282+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:59.326282+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3876,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31752192,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":31008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3902","reported_seq":"4489","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.667894+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:59.667894+0000","last_peered":"2021-01-15T18:42:59.667894+0000","last_clean":"2021-01-15T18:42:59.667894+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:59.667894+0000","last_undegraded":"2021-01-15T18:42:59.667894+0000","last_fullsized":"2021-01-15T18:42:59.667894+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3902,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31916198,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3900,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3770","reported_seq":"3977","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.715993+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:59.715993+0000","last_peered":"2021-01-15T18:42:59.715993+0000","last_clean":"2021-01-15T18:42:59.715993+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:59.715993+0000","last_undegraded":"2021-01-15T18:42:59.715993+0000","last_fullsized":"2021-01-15T18:42:59.715993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3770,"ondisk_log_size":3770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30851238,"num_objects":3767,"num_object_clones":0,"num_object_copies":15068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":30135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3813","reported_seq":"3867","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.405957+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:59.405957+0000","last_peered":"2021-01-15T18:42:59.405957+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:59.405957+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3813,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31195302,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2417}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3863","reported_seq":"3937","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.208178+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:42:58.208178+0000","last_peered":"2021-01-15T18:42:58.208178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:42:58.208178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3863,"ondisk_log_size":3863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31633491,"num_objects":3862,"num_object_clones":0,"num_object_copies":15448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3862,"num_write_kb":30893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3872","reported_seq":"3914","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.465029+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:59.465029+0000","last_peered":"2021-01-15T18:42:59.465029+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:59.465029+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3872,"ondisk_log_size":3872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31707219,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":30965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3864","reported_seq":"6799","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.358335+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:59.358335+0000","last_peered":"2021-01-15T18:42:59.358335+0000","last_clean":"2021-01-15T18:42:59.358335+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:59.358335+0000","last_undegraded":"2021-01-15T18:42:59.358335+0000","last_fullsized":"2021-01-15T18:42:59.358335+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3864,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31600889,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":30868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3750","reported_seq":"3807","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.317736+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:59.317736+0000","last_peered":"2021-01-15T18:42:59.317736+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:59.317736+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3750,"ondisk_log_size":3750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30687398,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3748,"num_write_kb":29975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3762","reported_seq":"3825","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.047205+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:42:59.047205+0000","last_peered":"2021-01-15T18:42:59.047205+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:42:59.047205+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3762,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30777510,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":30063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3775","reported_seq":"6154","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.163833+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:42:59.163833+0000","last_peered":"2021-01-15T18:42:59.163833+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:42:59.163833+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3775,"ondisk_log_size":3775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30867622,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3773,"num_write_kb":30156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":9453568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'3969","reported_seq":"4034","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.133122+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:42:59.133122+0000","last_peered":"2021-01-15T18:42:59.133122+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:42:59.133122+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3969,"ondisk_log_size":3969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32465062,"num_objects":3964,"num_object_clones":0,"num_object_copies":15856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3967,"num_write_kb":31716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3987","reported_seq":"4047","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.358771+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:59.358771+0000","last_peered":"2021-01-15T18:42:59.358771+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:59.358771+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3987,"ondisk_log_size":3987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32579916,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":31835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3788","reported_seq":"3869","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:58.181480+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:42:58.181480+0000","last_peered":"2021-01-15T18:42:58.181480+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:42:58.181480+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3788,"ondisk_log_size":3788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31006886,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2404}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":500762900,"num_objects":61142,"num_object_clones":0,"num_object_copies":244568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61169,"num_write_kb":489142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5596,"num_bytes_recovered":45547520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936558592,"data_stored":936558592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61196,"ondisk_log_size":61196,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376672,"kb_used_data":244888,"kb_used_omap":0,"kb_used_meta":131775,"kb_avail":93995168,"statfs":{"total":96636764160,"available":96251052032,"internally_reserved":0,"allocated":250765312,"data_stored":250570190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":134937652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,7,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.9710000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.1660000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":4.0800000000000001},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":4.3499999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.4420000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.2619999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362160,"kb_used_data":243272,"kb_used_omap":0,"kb_used_meta":118976,"kb_avail":94009680,"statfs":{"total":96636764160,"available":96265912320,"internally_reserved":0,"allocated":249110528,"data_stored":248915406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":121831424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.3479999999999999},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.96}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":9.0779999999999994},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":2.8340000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":2.4489999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280556,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370912,"kb_used_data":243672,"kb_used_omap":0,"kb_used_meta":127231,"kb_avail":94000928,"statfs":{"total":96636764160,"available":96256950272,"internally_reserved":0,"allocated":249520128,"data_stored":249325006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":130284596},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":3.7530000000000001},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.55800000000000005},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":16.77}]}]},{"osd":1,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284588,"kb_used_data":181668,"kb_used_omap":0,"kb_used_meta":102911,"kb_avail":94087252,"statfs":{"total":96636764160,"available":96345346048,"internally_reserved":0,"allocated":186028032,"data_stored":185832910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":105380916},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.996},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.4279999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.403},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":3.0169999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.8410000000000002},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.167}]}]}]}} 2021-01-15T18:43:03.071 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 183 20 62105 62085 2.64953 1.95312 0.0585512 0.0589582 2021-01-15T18:43:04.071 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 184 20 62411 62391 2.64812 2.39062 0.0452808 0.0589869 2021-01-15T18:43:05.071 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 185 20 62694 62674 2.64575 2.21094 0.160612 0.0590103 2021-01-15T18:43:05.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:06.071 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 186 20 62992 62972 2.64404 2.32812 0.103996 0.0590787 2021-01-15T18:43:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:06.274 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:06.341 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2021-01-15T18:43:04.438280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504924436,"num_objects":61650,"num_object_clones":0,"num_object_copies":246600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61677,"num_write_kb":493206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5765,"num_bytes_recovered":46931968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61704,"ondisk_log_size":61704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1409812,"kb_used_data":924228,"kb_used_omap":2,"kb_used_meta":485565,"kb_avail":376077548,"statfs":{"total":386547056640,"available":385103409152,"internally_reserved":0,"allocated":946409472,"data_stored":945628984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":497218716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,11,11,28,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27787264,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3392,"num_write_kb":27136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":3178496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015210"},"pg_stats":[{"pgid":"1.f","version":"36'3711","reported_seq":"3757","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.453409+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:42:59.453409+0000","last_peered":"2021-01-15T18:42:59.453409+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:42:59.453409+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3711,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30400512,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3711,"num_write_kb":29688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3746","reported_seq":"6665","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.678824+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:42:59.678824+0000","last_peered":"2021-01-15T18:42:59.678824+0000","last_clean":"2021-01-15T18:42:59.678824+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:42:59.678824+0000","last_undegraded":"2021-01-15T18:42:59.678824+0000","last_fullsized":"2021-01-15T18:42:59.678824+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3746,"ondisk_log_size":3746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30646438,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":29940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3828","reported_seq":"3907","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:03.278825+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:03.278825+0000","last_peered":"2021-01-15T18:43:03.278825+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:03.278825+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3828,"ondisk_log_size":3828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31330387,"num_objects":3825,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":30602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3876","reported_seq":"3922","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.326282+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:42:59.326282+0000","last_peered":"2021-01-15T18:42:59.326282+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:42:59.326282+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3876,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31752192,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":31008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2444}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3902","reported_seq":"4489","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.667894+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:42:59.667894+0000","last_peered":"2021-01-15T18:42:59.667894+0000","last_clean":"2021-01-15T18:42:59.667894+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:42:59.667894+0000","last_undegraded":"2021-01-15T18:42:59.667894+0000","last_fullsized":"2021-01-15T18:42:59.667894+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3902,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31916198,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3900,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3770","reported_seq":"3977","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.715993+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:42:59.715993+0000","last_peered":"2021-01-15T18:42:59.715993+0000","last_clean":"2021-01-15T18:42:59.715993+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:42:59.715993+0000","last_undegraded":"2021-01-15T18:42:59.715993+0000","last_fullsized":"2021-01-15T18:42:59.715993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3770,"ondisk_log_size":3770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30851238,"num_objects":3767,"num_object_clones":0,"num_object_copies":15068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":30135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3813","reported_seq":"3867","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.405957+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:42:59.405957+0000","last_peered":"2021-01-15T18:42:59.405957+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:42:59.405957+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3813,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31195302,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2417}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'3948","reported_seq":"4022","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:03.210262+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:03.210262+0000","last_peered":"2021-01-15T18:43:03.210262+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:03.210262+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3948,"ondisk_log_size":3948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32329811,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3947,"num_write_kb":31573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3872","reported_seq":"3914","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.465029+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:42:59.465029+0000","last_peered":"2021-01-15T18:42:59.465029+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:42:59.465029+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3872,"ondisk_log_size":3872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31707219,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":30965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2435}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3864","reported_seq":"6799","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:42:59.358335+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:42:59.358335+0000","last_peered":"2021-01-15T18:42:59.358335+0000","last_clean":"2021-01-15T18:42:59.358335+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:42:59.358335+0000","last_undegraded":"2021-01-15T18:42:59.358335+0000","last_fullsized":"2021-01-15T18:42:59.358335+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3864,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31600889,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":30868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3750","reported_seq":"3807","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.317736+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:42:59.317736+0000","last_peered":"2021-01-15T18:42:59.317736+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:42:59.317736+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3750,"ondisk_log_size":3750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30687398,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3748,"num_write_kb":29975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3849","reported_seq":"3912","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.012200+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:04.012200+0000","last_peered":"2021-01-15T18:43:04.012200+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:04.012200+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3849,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31490214,"num_objects":3845,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3854","reported_seq":"6570","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.162012+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:43:04.162012+0000","last_peered":"2021-01-15T18:43:04.162012+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:43:04.162012+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3854,"ondisk_log_size":3854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31514790,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1327,"num_bytes_recovered":10838016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4047","reported_seq":"4111","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.126904+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:04.126904+0000","last_peered":"2021-01-15T18:43:04.126904+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:04.126904+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4047,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33104038,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'3987","reported_seq":"4047","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:42:59.358771+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:42:59.358771+0000","last_peered":"2021-01-15T18:42:59.358771+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:42:59.358771+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3987,"ondisk_log_size":3987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32579916,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":31835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2493}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3887","reported_seq":"3968","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:03.096458+0000","last_change":"2021-01-15T18:40:32.747808+0000","last_active":"2021-01-15T18:43:03.096458+0000","last_peered":"2021-01-15T18:43:03.096458+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:03.096458+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3887,"ondisk_log_size":3887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31817894,"num_objects":3885,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504924436,"num_objects":61650,"num_object_clones":0,"num_object_copies":246600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61677,"num_write_kb":493206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5765,"num_bytes_recovered":46931968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":947478528,"data_stored":947478528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61704,"ondisk_log_size":61704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215146,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376672,"kb_used_data":244888,"kb_used_omap":0,"kb_used_meta":131775,"kb_avail":93995168,"statfs":{"total":96636764160,"available":96251052032,"internally_reserved":0,"allocated":250765312,"data_stored":250570190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":134937652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,7,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.9710000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.1660000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":4.0800000000000001},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":4.3499999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.4420000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.2619999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377848,"kb_used_data":248432,"kb_used_omap":0,"kb_used_meta":129408,"kb_avail":93993992,"statfs":{"total":96636764160,"available":96249847808,"internally_reserved":0,"allocated":254394368,"data_stored":254199246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":132513792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":8.2710000000000008},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.437}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":2.0899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.75},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":8.3940000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370704,"kb_used_data":249240,"kb_used_omap":0,"kb_used_meta":121471,"kb_avail":94001136,"statfs":{"total":96636764160,"available":96257163264,"internally_reserved":0,"allocated":255221760,"data_stored":255026638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":124386356},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,15],"upper_bound":64},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":1.119},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":1.4299999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.2320000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.851},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":8.625}]}]},{"osd":1,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284588,"kb_used_data":181668,"kb_used_omap":0,"kb_used_meta":102911,"kb_avail":94087252,"statfs":{"total":96636764160,"available":96345346048,"internally_reserved":0,"allocated":186028032,"data_stored":185832910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":105380916},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.996},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.4279999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.403},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":3.0169999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.8410000000000002},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.167}]}]}]}} 2021-01-15T18:43:06.342 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:07.072 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 187 20 63281 63261 2.64197 2.25781 0.0559435 0.0591266 2021-01-15T18:43:07.284 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:07.284 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15642/250004 objects degraded (6.257%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62501,"data_bytes":511895828,"bytes_used":1456291840,"bytes_avail":385090764800,"bytes_total":386547056640,"degraded_objects":15642,"degraded_total":250004,"degraded_ratio":0.062566998928017148,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":179339,"recovering_keys_per_sec":0,"num_objects_recovered":263,"num_bytes_recovered":2154496,"num_keys_recovered":0,"write_bytes_sec":2508028,"read_op_per_sec":0,"write_op_per_sec":306},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:07.338 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:07.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:08.072 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 188 20 63573 63553 2.64005 2.28125 0.0768041 0.0591699 2021-01-15T18:43:08.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:08.523 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:08.566 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2021-01-15T18:43:08.441255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":514263316,"num_objects":62790,"num_object_clones":0,"num_object_copies":251160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62817,"num_write_kb":502326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5771,"num_bytes_recovered":46981120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62844,"ondisk_log_size":62844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1422288,"kb_used_data":941380,"kb_used_omap":2,"kb_used_meta":480893,"kb_avail":376065072,"statfs":{"total":386547056640,"available":385090633728,"internally_reserved":0,"allocated":963973120,"data_stored":963192632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":492434588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,8,20,22,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":97,"apply_latency_ms":97,"commit_latency_ns":97000000,"apply_latency_ns":97000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25337856,"num_objects":3093,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":2203648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012445"},"pg_stats":[{"pgid":"1.f","version":"36'3792","reported_seq":"3838","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.485793+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:04.485793+0000","last_peered":"2021-01-15T18:43:04.485793+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:04.485793+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3792,"ondisk_log_size":3792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31064064,"num_objects":3792,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2438}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'3813","reported_seq":"6733","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:43:04.727200+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:04.727200+0000","last_peered":"2021-01-15T18:43:04.727200+0000","last_clean":"2021-01-15T18:43:04.727200+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:04.727200+0000","last_undegraded":"2021-01-15T18:43:04.727200+0000","last_fullsized":"2021-01-15T18:43:04.727200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3813,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31195302,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3926","reported_seq":"4005","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.245110+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:08.245110+0000","last_peered":"2021-01-15T18:43:08.245110+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:08.245110+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3926,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32133203,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"36'3968","reported_seq":"4012","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.522211+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:04.522211+0000","last_peered":"2021-01-15T18:43:04.522211+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:04.522211+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3968,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32505856,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"36'3989","reported_seq":"4577","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:43:04.646308+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:04.646308+0000","last_peered":"2021-01-15T18:43:04.646308+0000","last_clean":"2021-01-15T18:43:04.646308+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:04.646308+0000","last_undegraded":"2021-01-15T18:43:04.646308+0000","last_fullsized":"2021-01-15T18:43:04.646308+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3989,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32628902,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"36'3838","reported_seq":"4045","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:43:04.464897+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:04.464897+0000","last_peered":"2021-01-15T18:43:04.464897+0000","last_clean":"2021-01-15T18:43:04.464897+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:04.464897+0000","last_undegraded":"2021-01-15T18:43:04.464897+0000","last_fullsized":"2021-01-15T18:43:04.464897+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3838,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31408294,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3836,"num_write_kb":30679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'3891","reported_seq":"3944","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.462576+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:04.462576+0000","last_peered":"2021-01-15T18:43:04.462576+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:04.462576+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3891,"ondisk_log_size":3891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31834278,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":31100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2495}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4045","reported_seq":"4117","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.256044+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:08.256044+0000","last_peered":"2021-01-15T18:43:08.256044+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:08.256044+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4045,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33124435,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"36'3967","reported_seq":"4009","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.396616+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:04.396616+0000","last_peered":"2021-01-15T18:43:04.396616+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:04.396616+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3967,"ondisk_log_size":3967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32485459,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":31725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2530}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"36'3958","reported_seq":"6895","reported_epoch":"36","state":"active+clean","last_fresh":"2021-01-15T18:43:04.389241+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:04.389241+0000","last_peered":"2021-01-15T18:43:04.389241+0000","last_clean":"2021-01-15T18:43:04.389241+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:04.389241+0000","last_undegraded":"2021-01-15T18:43:04.389241+0000","last_fullsized":"2021-01-15T18:43:04.389241+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3958,"ondisk_log_size":3958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32370937,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3955,"num_write_kb":31620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"36'3834","reported_seq":"3892","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.507939+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:04.507939+0000","last_peered":"2021-01-15T18:43:04.507939+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:04.507939+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3834,"ondisk_log_size":3834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31375526,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":30647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"36'3849","reported_seq":"3912","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.012200+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:04.012200+0000","last_peered":"2021-01-15T18:43:04.012200+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:04.012200+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3849,"ondisk_log_size":3849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31490214,"num_objects":3845,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3847,"num_write_kb":30759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3854","reported_seq":"6570","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.162012+0000","last_change":"2021-01-15T18:42:00.450340+0000","last_active":"2021-01-15T18:43:04.162012+0000","last_peered":"2021-01-15T18:43:04.162012+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:43:04.162012+0000","last_undegraded":"2021-01-15T18:40:32.533936+0000","last_fullsized":"2021-01-15T18:40:32.487257+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3854,"ondisk_log_size":3854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31514790,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":30788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1327,"num_bytes_recovered":10838016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'4047","reported_seq":"4111","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.126904+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:04.126904+0000","last_peered":"2021-01-15T18:43:04.126904+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:04.126904+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4047,"ondisk_log_size":4047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33104038,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4045,"num_write_kb":32340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2600}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'4092","reported_seq":"4152","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:04.501560+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:04.501560+0000","last_peered":"2021-01-15T18:43:04.501560+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:04.501560+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4092,"ondisk_log_size":4092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33440076,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4089,"num_write_kb":32675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3981","reported_seq":"4075","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.291787+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:08.291787+0000","last_peered":"2021-01-15T18:43:08.291787+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:08.291787+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3981,"ondisk_log_size":3981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32587942,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":31826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":514263316,"num_objects":62790,"num_object_clones":0,"num_object_copies":251160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62817,"num_write_kb":502326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5771,"num_bytes_recovered":46981120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":965402624,"data_stored":965402624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62844,"ondisk_log_size":62844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215147,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372248,"kb_used_data":250320,"kb_used_omap":0,"kb_used_meta":121919,"kb_avail":93999592,"statfs":{"total":96636764160,"available":96255582208,"internally_reserved":0,"allocated":256327680,"data_stored":256132558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":124845108},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,12,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.153}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":19.652999999999999},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":19.603000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.944},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.4109999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377976,"kb_used_data":254196,"kb_used_omap":0,"kb_used_meta":123776,"kb_avail":93993864,"statfs":{"total":96636764160,"available":96249716736,"internally_reserved":0,"allocated":260296704,"data_stored":260101582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":126746624},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.5},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":3.4590000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370704,"kb_used_data":249240,"kb_used_omap":0,"kb_used_meta":121471,"kb_avail":94001136,"statfs":{"total":96636764160,"available":96257163264,"internally_reserved":0,"allocated":255221760,"data_stored":255026638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":124386356},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,15],"upper_bound":64},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":1.119},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":1.4299999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.2320000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.851},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":8.625}]}]},{"osd":1,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301360,"kb_used_data":187624,"kb_used_omap":0,"kb_used_meta":113727,"kb_avail":94070480,"statfs":{"total":96636764160,"available":96328171520,"internally_reserved":0,"allocated":192126976,"data_stored":191931854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":116456500},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":4.1260000000000003},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.75600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.403},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":3.0169999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.8340000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":3.5430000000000001}]}]}]}} 2021-01-15T18:43:09.072 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 189 20 63835 63815 2.63691 2.04688 0.0193374 0.0591905 2021-01-15T18:43:10.074 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 190 20 64047 64027 2.63172 1.65625 0.0608277 0.059356 2021-01-15T18:43:11.075 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 191 20 64346 64326 2.63016 2.33594 0.071965 0.0593936 2021-01-15T18:43:11.570 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:12.075 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 192 20 64628 64608 2.62793 2.20312 0.102921 0.0594401 2021-01-15T18:43:12.696 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:12.712 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:12.767 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2021-01-15T18:43:12.442270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523143444,"num_objects":63874,"num_object_clones":0,"num_object_copies":255496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63901,"num_write_kb":510998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5853,"num_bytes_recovered":47652864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63928,"ondisk_log_size":63928,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1455436,"kb_used_data":958640,"kb_used_omap":2,"kb_used_meta":496765,"kb_avail":376031924,"statfs":{"total":386547056640,"available":385056690176,"internally_reserved":0,"allocated":981647360,"data_stored":980857402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":508687516},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,11,19,14,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":99,"apply_latency_ms":99,"commit_latency_ns":99000000,"apply_latency_ns":99000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18219008,"num_objects":2224,"num_object_clones":0,"num_object_copies":8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003989"},"pg_stats":[{"pgid":"1.f","version":"37'3857","reported_seq":"3905","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.446514+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:09.446514+0000","last_peered":"2021-01-15T18:43:09.446514+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:09.446514+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3857,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31596544,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":30856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'3905","reported_seq":"6825","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.725778+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:09.725778+0000","last_peered":"2021-01-15T18:43:09.725778+0000","last_clean":"2021-01-15T18:43:09.725778+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:09.725778+0000","last_undegraded":"2021-01-15T18:43:09.725778+0000","last_fullsized":"2021-01-15T18:43:09.725778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3905,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31948966,"num_objects":3901,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"36'3926","reported_seq":"4005","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.245110+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:08.245110+0000","last_peered":"2021-01-15T18:43:08.245110+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:08.245110+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3926,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32133203,"num_objects":3923,"num_object_clones":0,"num_object_copies":15692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":31386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"37'4063","reported_seq":"4111","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.451307+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:09.451307+0000","last_peered":"2021-01-15T18:43:09.451307+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:09.451307+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4063,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33284096,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4074","reported_seq":"4664","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.765723+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:09.765723+0000","last_peered":"2021-01-15T18:43:09.765723+0000","last_clean":"2021-01-15T18:43:09.765723+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:09.765723+0000","last_undegraded":"2021-01-15T18:43:09.765723+0000","last_fullsized":"2021-01-15T18:43:09.765723+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4074,"ondisk_log_size":4074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33325222,"num_objects":4069,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'3916","reported_seq":"4124","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.540722+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:09.540722+0000","last_peered":"2021-01-15T18:43:09.540722+0000","last_clean":"2021-01-15T18:43:09.540722+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:09.540722+0000","last_undegraded":"2021-01-15T18:43:09.540722+0000","last_fullsized":"2021-01-15T18:43:09.540722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3916,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32047270,"num_objects":3913,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":31303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'3972","reported_seq":"4028","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.490389+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:09.490389+0000","last_peered":"2021-01-15T18:43:09.490389+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:09.490389+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3972,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32497830,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2576}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"36'4045","reported_seq":"4117","reported_epoch":"36","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.256044+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:08.256044+0000","last_peered":"2021-01-15T18:43:08.256044+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:08.256044+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4045,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33124435,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2592}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"37'4030","reported_seq":"4074","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.413274+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:09.413274+0000","last_peered":"2021-01-15T18:43:09.413274+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:09.413274+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4030,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33001555,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"37'4042","reported_seq":"6981","reported_epoch":"37","state":"active+clean","last_fresh":"2021-01-15T18:43:09.476091+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:09.476091+0000","last_peered":"2021-01-15T18:43:09.476091+0000","last_clean":"2021-01-15T18:43:09.476091+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:09.476091+0000","last_undegraded":"2021-01-15T18:43:09.476091+0000","last_fullsized":"2021-01-15T18:43:09.476091+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4042,"ondisk_log_size":4042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33059065,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":32292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"37'3928","reported_seq":"3988","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.531300+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:09.531300+0000","last_peered":"2021-01-15T18:43:09.531300+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:09.531300+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3928,"ondisk_log_size":3928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32145574,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"37'3939","reported_seq":"4004","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.879080+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:08.879080+0000","last_peered":"2021-01-15T18:43:08.879080+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:08.879080+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3939,"ondisk_log_size":3939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32227494,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"36'3939","reported_seq":"6822","reported_epoch":"37","state":"peering","last_fresh":"2021-01-15T18:43:08.550341+0000","last_change":"2021-01-15T18:43:08.552003+0000","last_active":"2021-01-15T18:43:08.550341+0000","last_peered":"2021-01-15T18:43:08.514536+0000","last_clean":"2021-01-15T18:39:35.050518+0000","last_became_active":"2021-01-15T18:40:32.749155+0000","last_became_peered":"2021-01-15T18:40:32.749155+0000","last_unstale":"2021-01-15T18:43:08.550341+0000","last_undegraded":"2021-01-15T18:43:08.550341+0000","last_fullsized":"2021-01-15T18:43:08.550341+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3939,"ondisk_log_size":3939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32211110,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":31468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'4140","reported_seq":"4206","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.114565+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:09.114565+0000","last_peered":"2021-01-15T18:43:09.114565+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:09.114565+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4140,"ondisk_log_size":4140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33865894,"num_objects":4135,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":33084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'4171","reported_seq":"4233","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.512280+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:09.512280+0000","last_peered":"2021-01-15T18:43:09.512280+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:09.512280+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4171,"ondisk_log_size":4171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34087244,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4168,"num_write_kb":33307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'3981","reported_seq":"4075","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:08.291787+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:08.291787+0000","last_peered":"2021-01-15T18:43:08.291787+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:08.291787+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3981,"ondisk_log_size":3981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32587942,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":31826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523143444,"num_objects":63874,"num_object_clones":0,"num_object_copies":255496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63901,"num_write_kb":510998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5853,"num_bytes_recovered":47652864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":982470656,"data_stored":982470656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63928,"ondisk_log_size":63928,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388188,"kb_used_data":255700,"kb_used_omap":0,"kb_used_meta":132479,"kb_avail":93983652,"statfs":{"total":96636764160,"available":96239259648,"internally_reserved":0,"allocated":261836800,"data_stored":261636943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":135658548},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,12,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.153}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":19.652999999999999},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":19.603000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.944},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.4109999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377976,"kb_used_data":254196,"kb_used_omap":0,"kb_used_meta":123776,"kb_avail":93993864,"statfs":{"total":96636764160,"available":96249716736,"internally_reserved":0,"allocated":260296704,"data_stored":260101582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":126746624},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.5},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":3.4590000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387308,"kb_used_data":255076,"kb_used_omap":0,"kb_used_meta":132223,"kb_avail":93984532,"statfs":{"total":96636764160,"available":96240160768,"internally_reserved":0,"allocated":261197824,"data_stored":261002702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":135396404},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":1.1599999999999999},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":9.4499999999999993}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":1.6419999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.2389999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":3.0409999999999999},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":3.141}]}]},{"osd":1,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301964,"kb_used_data":193668,"kb_used_omap":0,"kb_used_meta":108287,"kb_avail":94069876,"statfs":{"total":96636764160,"available":96327553024,"internally_reserved":0,"allocated":198316032,"data_stored":198116175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":110885940},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.4160000000000004},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.5409999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.425}]}]}]}} 2021-01-15T18:43:12.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:13.078 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 193 20 64906 64886 2.62554 2.17188 0.0549892 0.0594946 2021-01-15T18:43:14.078 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 194 20 65181 65161 2.62308 2.14844 0.0676508 0.0595523 2021-01-15T18:43:14.098 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:14.098 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15552/255496 objects degraded (6.087%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":244,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63874,"data_bytes":523143444,"bytes_used":1490366464,"bytes_avail":385056690176,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":15552,"degraded_total":255496,"degraded_ratio":0.060869837492563486,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":90067,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"write_bytes_sec":2276240,"read_op_per_sec":0,"write_op_per_sec":277},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:14.169 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:14.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:15.079 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 195 20 65453 65433 2.62052 2.125 0.0634779 0.0596079 2021-01-15T18:43:15.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:15.439 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:15.480 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2021-01-15T18:43:14.442746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527370516,"num_objects":64390,"num_object_clones":0,"num_object_copies":257560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64417,"num_write_kb":515126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":48275456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64444,"ondisk_log_size":64444,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1470984,"kb_used_data":969392,"kb_used_omap":2,"kb_used_meta":501629,"kb_avail":376016376,"statfs":{"total":386547056640,"available":385040769024,"internally_reserved":0,"allocated":992657408,"data_stored":991848468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":513668252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,17,26,11,14,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15474688,"num_objects":1889,"num_object_clones":0,"num_object_copies":7556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1889,"num_write_kb":15112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003978"},"pg_stats":[{"pgid":"1.f","version":"37'3857","reported_seq":"3905","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.446514+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:09.446514+0000","last_peered":"2021-01-15T18:43:09.446514+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:09.446514+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3857,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31596544,"num_objects":3857,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":30856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'3905","reported_seq":"6825","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.725778+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:09.725778+0000","last_peered":"2021-01-15T18:43:09.725778+0000","last_clean":"2021-01-15T18:43:09.725778+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:09.725778+0000","last_undegraded":"2021-01-15T18:43:09.725778+0000","last_fullsized":"2021-01-15T18:43:09.725778+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3905,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31948966,"num_objects":3901,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":31212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4017","reported_seq":"4098","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:13.177987+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:13.177987+0000","last_peered":"2021-01-15T18:43:13.177987+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:13.177987+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4017,"ondisk_log_size":4017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32878675,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4016,"num_write_kb":32114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2598}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"37'4063","reported_seq":"4111","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.451307+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:09.451307+0000","last_peered":"2021-01-15T18:43:09.451307+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:09.451307+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4063,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33284096,"num_objects":4063,"num_object_clones":0,"num_object_copies":16252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":32504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4074","reported_seq":"4664","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.765723+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:09.765723+0000","last_peered":"2021-01-15T18:43:09.765723+0000","last_clean":"2021-01-15T18:43:09.765723+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:09.765723+0000","last_undegraded":"2021-01-15T18:43:09.765723+0000","last_fullsized":"2021-01-15T18:43:09.765723+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4074,"ondisk_log_size":4074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33325222,"num_objects":4069,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":32556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'3916","reported_seq":"4124","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:09.540722+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:09.540722+0000","last_peered":"2021-01-15T18:43:09.540722+0000","last_clean":"2021-01-15T18:43:09.540722+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:09.540722+0000","last_undegraded":"2021-01-15T18:43:09.540722+0000","last_fullsized":"2021-01-15T18:43:09.540722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3916,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32047270,"num_objects":3913,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":31303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'3972","reported_seq":"4028","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.490389+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:09.490389+0000","last_peered":"2021-01-15T18:43:09.490389+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:09.490389+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3972,"ondisk_log_size":3972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32497830,"num_objects":3968,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2576}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4131","reported_seq":"4207","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:13.137226+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:13.137226+0000","last_peered":"2021-01-15T18:43:13.137226+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:13.137226+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4131,"ondisk_log_size":4131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33828947,"num_objects":4130,"num_object_clones":0,"num_object_copies":16520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4130,"num_write_kb":33037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"37'4030","reported_seq":"4074","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.413274+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:09.413274+0000","last_peered":"2021-01-15T18:43:09.413274+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:09.413274+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4030,"ondisk_log_size":4030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33001555,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":32229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"37'4042","reported_seq":"6981","reported_epoch":"37","state":"active+clean","last_fresh":"2021-01-15T18:43:09.476091+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:09.476091+0000","last_peered":"2021-01-15T18:43:09.476091+0000","last_clean":"2021-01-15T18:43:09.476091+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:09.476091+0000","last_undegraded":"2021-01-15T18:43:09.476091+0000","last_fullsized":"2021-01-15T18:43:09.476091+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4042,"ondisk_log_size":4042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33059065,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":32292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"37'3928","reported_seq":"3988","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.531300+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:09.531300+0000","last_peered":"2021-01-15T18:43:09.531300+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:09.531300+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3928,"ondisk_log_size":3928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32145574,"num_objects":3925,"num_object_clones":0,"num_object_copies":15700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":31399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4027","reported_seq":"4093","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.123022+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:14.123022+0000","last_peered":"2021-01-15T18:43:14.123022+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:14.123022+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4027,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32948390,"num_objects":4023,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4036","reported_seq":"6928","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.043185+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:14.043185+0000","last_peered":"2021-01-15T18:43:14.043185+0000","last_clean":"2021-01-15T18:43:14.043185+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:14.043185+0000","last_undegraded":"2021-01-15T18:43:14.043185+0000","last_fullsized":"2021-01-15T18:43:14.043185+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33005734,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4212","reported_seq":"4278","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.163412+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:14.163412+0000","last_peered":"2021-01-15T18:43:14.163412+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:14.163412+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4212,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34455718,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'4171","reported_seq":"4233","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:09.512280+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:09.512280+0000","last_peered":"2021-01-15T18:43:09.512280+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:09.512280+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4171,"ondisk_log_size":4171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34087244,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4168,"num_write_kb":33307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4063","reported_seq":"4311","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:13.312796+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:13.312796+0000","last_peered":"2021-01-15T18:43:13.312796+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:13.312796+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4063,"ondisk_log_size":4063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33259686,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":32482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":527370516,"num_objects":64390,"num_object_clones":0,"num_object_copies":257560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64417,"num_write_kb":515126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":48275456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993529856,"data_stored":993529856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64444,"ondisk_log_size":64444,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388188,"kb_used_data":255700,"kb_used_omap":0,"kb_used_meta":132479,"kb_avail":93983652,"statfs":{"total":96636764160,"available":96239259648,"internally_reserved":0,"allocated":261836800,"data_stored":261636943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":135658548},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,12,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.153}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":19.652999999999999},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":19.603000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.944},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.4109999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394076,"kb_used_data":259668,"kb_used_omap":0,"kb_used_meta":134400,"kb_avail":93977764,"statfs":{"total":96636764160,"available":96233230336,"internally_reserved":0,"allocated":265900032,"data_stored":265695419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":137625600},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,7,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.88},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":4.2199999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":4.7439999999999998},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":3.7189999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":3.79},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":0.77000000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280560,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386756,"kb_used_data":260356,"kb_used_omap":0,"kb_used_meta":126463,"kb_avail":93985084,"statfs":{"total":96636764160,"available":96240726016,"internally_reserved":0,"allocated":266604544,"data_stored":266399931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":129498164},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":11.471}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.92600000000000005},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.97199999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.8339999999999996},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":56.927}]}]},{"osd":1,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301964,"kb_used_data":193668,"kb_used_omap":0,"kb_used_meta":108287,"kb_avail":94069876,"statfs":{"total":96636764160,"available":96327553024,"internally_reserved":0,"allocated":198316032,"data_stored":198116175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":110885940},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.4160000000000004},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.5409999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":1.425}]}]}]}} 2021-01-15T18:43:16.081 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 196 20 65745 65725 2.61876 2.28125 0.0579898 0.0596494 2021-01-15T18:43:17.081 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 197 20 66025 66005 2.61657 2.1875 0.0559486 0.0597013 2021-01-15T18:43:18.081 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 198 20 66319 66299 2.61495 2.29688 0.125169 0.0597339 2021-01-15T18:43:18.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:19.082 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 199 20 66598 66578 2.61275 2.17969 0.100274 0.0597893 2021-01-15T18:43:19.665 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:19.712 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:19.729 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2021-01-15T18:43:18.443757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":536783124,"num_objects":65539,"num_object_clones":0,"num_object_copies":262156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65566,"num_write_kb":524318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5993,"num_bytes_recovered":48799744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65593,"ondisk_log_size":65593,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1487416,"kb_used_data":986472,"kb_used_omap":2,"kb_used_meta":500989,"kb_avail":375999944,"statfs":{"total":386547056640,"available":385023942656,"internally_reserved":0,"allocated":1010147328,"data_stored":1009328876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":513012892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,10,13,24,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22519808,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":21992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.911538"},"pg_stats":[{"pgid":"1.f","version":"38'3966","reported_seq":"4012","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.529686+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:14.529686+0000","last_peered":"2021-01-15T18:43:14.529686+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:14.529686+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3966,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32489472,"num_objects":3966,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":31728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2612}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'3994","reported_seq":"6915","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.734484+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:14.734484+0000","last_peered":"2021-01-15T18:43:14.734484+0000","last_clean":"2021-01-15T18:43:14.734484+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:14.734484+0000","last_undegraded":"2021-01-15T18:43:14.734484+0000","last_fullsized":"2021-01-15T18:43:14.734484+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":3994,"ondisk_log_size":3994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32678054,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":31924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4089","reported_seq":"4169","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.282858+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:18.282858+0000","last_peered":"2021-01-15T18:43:18.282858+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:18.282858+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4089,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33468499,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4088,"num_write_kb":32690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4149","reported_seq":"4197","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.505684+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:14.505684+0000","last_peered":"2021-01-15T18:43:14.505684+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:14.505684+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4149,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33988608,"num_objects":4149,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":33192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4149","reported_seq":"4738","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.781233+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:14.781233+0000","last_peered":"2021-01-15T18:43:14.781233+0000","last_clean":"2021-01-15T18:43:14.781233+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:14.781233+0000","last_undegraded":"2021-01-15T18:43:14.781233+0000","last_fullsized":"2021-01-15T18:43:14.781233+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4149,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33939622,"num_objects":4144,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4012","reported_seq":"4221","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.783525+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:14.783525+0000","last_peered":"2021-01-15T18:43:14.783525+0000","last_clean":"2021-01-15T18:43:14.783525+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:14.783525+0000","last_undegraded":"2021-01-15T18:43:14.783525+0000","last_fullsized":"2021-01-15T18:43:14.783525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4012,"ondisk_log_size":4012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32833702,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":32071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4055","reported_seq":"4111","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.449404+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:14.449404+0000","last_peered":"2021-01-15T18:43:14.449404+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:14.449404+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4055,"ondisk_log_size":4055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33177766,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4053,"num_write_kb":32412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2659}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4229","reported_seq":"4305","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.074698+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:18.074698+0000","last_peered":"2021-01-15T18:43:18.074698+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:18.074698+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4229,"ondisk_log_size":4229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34631763,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":33821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4132","reported_seq":"4176","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.498899+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:14.498899+0000","last_peered":"2021-01-15T18:43:14.498899+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:14.498899+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4132,"ondisk_log_size":4132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33837139,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":33045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2695}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4127","reported_seq":"7066","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.346843+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:14.346843+0000","last_peered":"2021-01-15T18:43:14.346843+0000","last_clean":"2021-01-15T18:43:14.346843+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:14.346843+0000","last_undegraded":"2021-01-15T18:43:14.346843+0000","last_fullsized":"2021-01-15T18:43:14.346843+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4127,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33755385,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4012","reported_seq":"4072","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.313947+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:14.313947+0000","last_peered":"2021-01-15T18:43:14.313947+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:14.313947+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4012,"ondisk_log_size":4012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32833702,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":32071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4027","reported_seq":"4093","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.123022+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:14.123022+0000","last_peered":"2021-01-15T18:43:14.123022+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:14.123022+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4027,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32948390,"num_objects":4023,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4036","reported_seq":"6928","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:14.043185+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:14.043185+0000","last_peered":"2021-01-15T18:43:14.043185+0000","last_clean":"2021-01-15T18:43:14.043185+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:14.043185+0000","last_undegraded":"2021-01-15T18:43:14.043185+0000","last_fullsized":"2021-01-15T18:43:14.043185+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33005734,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4212","reported_seq":"4278","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.163412+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:14.163412+0000","last_peered":"2021-01-15T18:43:14.163412+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:14.163412+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4212,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34455718,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4255","reported_seq":"4317","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:14.525473+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:14.525473+0000","last_peered":"2021-01-15T18:43:14.525473+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:14.525473+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4255,"ondisk_log_size":4255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34775372,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":33979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4149","reported_seq":"4525","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.296852+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:18.296852+0000","last_peered":"2021-01-15T18:43:18.296852+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:18.296852+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4149,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33964198,"num_objects":4147,"num_object_clones":0,"num_object_copies":16588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":536783124,"num_objects":65539,"num_object_clones":0,"num_object_copies":262156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65566,"num_write_kb":524318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5993,"num_bytes_recovered":48799744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011134464,"data_stored":1011134464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65593,"ondisk_log_size":65593,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388104,"kb_used_data":261316,"kb_used_omap":0,"kb_used_meta":126783,"kb_avail":93983736,"statfs":{"total":96636764160,"available":96239345664,"internally_reserved":0,"allocated":267587584,"data_stored":267382971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":129825844},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,4,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.008},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.0940000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":1.3640000000000001},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":1.4399999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.6640000000000001},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.91500000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393896,"kb_used_data":265252,"kb_used_omap":0,"kb_used_meta":128640,"kb_avail":93977944,"statfs":{"total":96636764160,"available":96233414656,"internally_reserved":0,"allocated":271618048,"data_stored":271413435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":131727360},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,15,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":4.4050000000000002},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.2189999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":3.0499999999999998},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":2.2989999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280560,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386756,"kb_used_data":260356,"kb_used_omap":0,"kb_used_meta":126463,"kb_avail":93985084,"statfs":{"total":96636764160,"available":96240726016,"internally_reserved":0,"allocated":266604544,"data_stored":266399931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":129498164},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":11.471}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.92600000000000005},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.97199999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.8339999999999996},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":56.927}]}]},{"osd":1,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318660,"kb_used_data":199548,"kb_used_omap":0,"kb_used_meta":119103,"kb_avail":94053180,"statfs":{"total":96636764160,"available":96310456320,"internally_reserved":0,"allocated":204337152,"data_stored":204132539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":121961524},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":27.341000000000001},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":12.824}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.6100000000000003},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":4.3150000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":4.4050000000000002},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":4.2190000000000003}]}]}]}} 2021-01-15T18:43:19.731 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:20.082 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:43:20.082384+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0598371 2021-01-15T18:43:20.082 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:43:20.083 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 200 20 66877 66857 2.61059 2.17969 0.0979724 0.0598371 2021-01-15T18:43:20.990 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:20.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15412/266712 objects degraded (5.779%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":251,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66678,"data_bytes":546113812,"bytes_used":1557450752,"bytes_avail":384989605888,"bytes_total":386547056640,"degraded_objects":15412,"degraded_total":266712,"degraded_ratio":0.057785176519991592,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105079,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":2717057,"read_op_per_sec":0,"write_op_per_sec":331},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:21.081 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:21.082 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:21.083 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 201 20 67139 67119 2.60778 2.04688 0.0743155 0.0599009 2021-01-15T18:43:22.082 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 202 20 67414 67394 2.60551 2.14844 0.0985392 0.059953 2021-01-15T18:43:22.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:22.207 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:22.220 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2021-01-15T18:43:20.444278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":546113812,"num_objects":66678,"num_object_clones":0,"num_object_copies":266712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66705,"num_write_kb":533430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5993,"num_bytes_recovered":48799744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66732,"ondisk_log_size":66732,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1520948,"kb_used_data":1004056,"kb_used_omap":2,"kb_used_meta":516861,"kb_avail":375966412,"statfs":{"total":386547056640,"available":384989605888,"internally_reserved":0,"allocated":1028153344,"data_stored":1027334892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":529265820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,25,26,19,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29655040,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.914395"},"pg_stats":[{"pgid":"1.f","version":"38'4060","reported_seq":"4108","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.403644+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:19.403644+0000","last_peered":"2021-01-15T18:43:19.403644+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:19.403644+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4060,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33259520,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":32480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4079","reported_seq":"7001","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.685911+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:19.685911+0000","last_peered":"2021-01-15T18:43:19.685911+0000","last_clean":"2021-01-15T18:43:19.685911+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:19.685911+0000","last_undegraded":"2021-01-15T18:43:19.685911+0000","last_fullsized":"2021-01-15T18:43:19.685911+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4079,"ondisk_log_size":4079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33374374,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4089","reported_seq":"4169","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.282858+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:18.282858+0000","last_peered":"2021-01-15T18:43:18.282858+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:18.282858+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4089,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33468499,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4088,"num_write_kb":32690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4225","reported_seq":"4273","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.438778+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:19.438778+0000","last_peered":"2021-01-15T18:43:19.438778+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:19.438778+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4225,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34611200,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4243","reported_seq":"4833","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.718541+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:19.718541+0000","last_peered":"2021-01-15T18:43:19.718541+0000","last_clean":"2021-01-15T18:43:19.718541+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:19.718541+0000","last_undegraded":"2021-01-15T18:43:19.718541+0000","last_fullsized":"2021-01-15T18:43:19.718541+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4243,"ondisk_log_size":4243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34709670,"num_objects":4238,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4092","reported_seq":"4301","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.537295+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:19.537295+0000","last_peered":"2021-01-15T18:43:19.537295+0000","last_clean":"2021-01-15T18:43:19.537295+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:19.537295+0000","last_undegraded":"2021-01-15T18:43:19.537295+0000","last_fullsized":"2021-01-15T18:43:19.537295+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4092,"ondisk_log_size":4092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33489062,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4129","reported_seq":"4185","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.338126+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:19.338126+0000","last_peered":"2021-01-15T18:43:19.338126+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:19.338126+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4129,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33783974,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2733}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4229","reported_seq":"4305","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.074698+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:18.074698+0000","last_peered":"2021-01-15T18:43:18.074698+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:18.074698+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4229,"ondisk_log_size":4229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34631763,"num_objects":4228,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":33821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2776}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4217","reported_seq":"4261","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.466442+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:19.466442+0000","last_peered":"2021-01-15T18:43:19.466442+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:19.466442+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4217,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34533459,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2780}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4226","reported_seq":"7165","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.534961+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:19.534961+0000","last_peered":"2021-01-15T18:43:19.534961+0000","last_clean":"2021-01-15T18:43:19.534961+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:19.534961+0000","last_undegraded":"2021-01-15T18:43:19.534961+0000","last_fullsized":"2021-01-15T18:43:19.534961+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4226,"ondisk_log_size":4226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34566393,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4109","reported_seq":"4169","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.523821+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:19.523821+0000","last_peered":"2021-01-15T18:43:19.523821+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:19.523821+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4109,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33628326,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4112","reported_seq":"4176","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.183527+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:19.183527+0000","last_peered":"2021-01-15T18:43:19.183527+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:19.183527+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4112,"ondisk_log_size":4112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33644710,"num_objects":4108,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":32863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4126","reported_seq":"7019","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.110984+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:19.110984+0000","last_peered":"2021-01-15T18:43:19.110984+0000","last_clean":"2021-01-15T18:43:19.110984+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:19.110984+0000","last_undegraded":"2021-01-15T18:43:19.110984+0000","last_fullsized":"2021-01-15T18:43:19.110984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4126,"ondisk_log_size":4126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33743014,"num_objects":4120,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":32964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4293","reported_seq":"4360","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.184390+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:19.184390+0000","last_peered":"2021-01-15T18:43:19.184390+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:19.184390+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4293,"ondisk_log_size":4293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35119270,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2846}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4354","reported_seq":"4416","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.440670+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:19.440670+0000","last_peered":"2021-01-15T18:43:19.440670+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:19.440670+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4354,"ondisk_log_size":4354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35586380,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":34771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4149","reported_seq":"4525","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:18.296852+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:18.296852+0000","last_peered":"2021-01-15T18:43:18.296852+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:18.296852+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4149,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33964198,"num_objects":4147,"num_object_clones":0,"num_object_copies":16588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2911}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":546113812,"num_objects":66678,"num_object_clones":0,"num_object_copies":266712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66705,"num_write_kb":533430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5993,"num_bytes_recovered":48799744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028890624,"data_stored":1028890624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66732,"ondisk_log_size":66732,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404492,"kb_used_data":267012,"kb_used_omap":0,"kb_used_meta":137471,"kb_avail":93967348,"statfs":{"total":96636764160,"available":96222564352,"internally_reserved":0,"allocated":273420288,"data_stored":273215675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":140770356},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,12,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.8900000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.0979999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":5.9619999999999997},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":5.9039999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.0510000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.032}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393896,"kb_used_data":265252,"kb_used_omap":0,"kb_used_meta":128640,"kb_avail":93977944,"statfs":{"total":96636764160,"available":96233414656,"internally_reserved":0,"allocated":271618048,"data_stored":271413435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":131727360},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,15,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":4.4050000000000002},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.2189999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":3.0499999999999998},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":2.2989999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280561,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403432,"kb_used_data":266212,"kb_used_omap":0,"kb_used_meta":137215,"kb_avail":93968408,"statfs":{"total":96636764160,"available":96223649792,"internally_reserved":0,"allocated":272601088,"data_stored":272396475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":140508212},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":10.643000000000001},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":22.099}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":5.4720000000000004},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.9580000000000002},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":3.0569999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319128,"kb_used_data":205580,"kb_used_omap":0,"kb_used_meta":113535,"kb_avail":94052712,"statfs":{"total":96636764160,"available":96309977088,"internally_reserved":0,"allocated":210513920,"data_stored":210309307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":116259892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":6.9569999999999999},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":2.4969999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":6.6849999999999996},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":6.6509999999999998},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":2.6000000000000001}]}]}]}} 2021-01-15T18:43:23.085 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 203 20 67673 67653 2.60262 2.02344 0.0679297 0.0600208 2021-01-15T18:43:24.086 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 204 20 67958 67938 2.60077 2.22656 0.0638318 0.0600638 2021-01-15T18:43:25.089 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 205 20 68239 68219 2.59878 2.19531 0.062095 0.0601091 2021-01-15T18:43:25.222 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:26.086 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 206 20 68514 68494 2.59659 2.14844 0.0751259 0.0601617 2021-01-15T18:43:26.168 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:26.182 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2021-01-15T18:43:24.451302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":550144276,"num_objects":67170,"num_object_clones":0,"num_object_copies":268680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67197,"num_write_kb":537366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6069,"num_bytes_recovered":49422336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67224,"ondisk_log_size":67224,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1536636,"kb_used_data":1014952,"kb_used_omap":2,"kb_used_meta":521661,"kb_avail":375950724,"statfs":{"total":386547056640,"available":384973541376,"internally_reserved":0,"allocated":1039310848,"data_stored":1038492396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":534181020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,27,17,20,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":103,"apply_latency_ms":103,"commit_latency_ns":103000000,"apply_latency_ns":103000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27000832,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":1769472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009032"},"pg_stats":[{"pgid":"1.f","version":"38'4060","reported_seq":"4108","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.403644+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:19.403644+0000","last_peered":"2021-01-15T18:43:19.403644+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:19.403644+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4060,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33259520,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":32480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4079","reported_seq":"7001","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.685911+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:19.685911+0000","last_peered":"2021-01-15T18:43:19.685911+0000","last_clean":"2021-01-15T18:43:19.685911+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:19.685911+0000","last_undegraded":"2021-01-15T18:43:19.685911+0000","last_fullsized":"2021-01-15T18:43:19.685911+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4079,"ondisk_log_size":4079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33374374,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4175","reported_seq":"4255","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:23.197046+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:23.197046+0000","last_peered":"2021-01-15T18:43:23.197046+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:23.197046+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4175,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34173011,"num_objects":4172,"num_object_clones":0,"num_object_copies":16688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":33378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4225","reported_seq":"4273","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.438778+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:19.438778+0000","last_peered":"2021-01-15T18:43:19.438778+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:19.438778+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4225,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34611200,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4243","reported_seq":"4833","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.718541+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:19.718541+0000","last_peered":"2021-01-15T18:43:19.718541+0000","last_clean":"2021-01-15T18:43:19.718541+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:19.718541+0000","last_undegraded":"2021-01-15T18:43:19.718541+0000","last_fullsized":"2021-01-15T18:43:19.718541+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4243,"ondisk_log_size":4243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34709670,"num_objects":4238,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4092","reported_seq":"4301","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.537295+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:19.537295+0000","last_peered":"2021-01-15T18:43:19.537295+0000","last_clean":"2021-01-15T18:43:19.537295+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:19.537295+0000","last_undegraded":"2021-01-15T18:43:19.537295+0000","last_fullsized":"2021-01-15T18:43:19.537295+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4092,"ondisk_log_size":4092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33489062,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":32711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4129","reported_seq":"4185","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.338126+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:19.338126+0000","last_peered":"2021-01-15T18:43:19.338126+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:19.338126+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4129,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33783974,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2733}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4310","reported_seq":"4386","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:23.311923+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:23.311923+0000","last_peered":"2021-01-15T18:43:23.311923+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:23.311923+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4310,"ondisk_log_size":4310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35295315,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":34469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4217","reported_seq":"4261","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.466442+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:19.466442+0000","last_peered":"2021-01-15T18:43:19.466442+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:19.466442+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4217,"ondisk_log_size":4217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34533459,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":33725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2780}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4226","reported_seq":"7165","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:19.534961+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:19.534961+0000","last_peered":"2021-01-15T18:43:19.534961+0000","last_clean":"2021-01-15T18:43:19.534961+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:19.534961+0000","last_undegraded":"2021-01-15T18:43:19.534961+0000","last_fullsized":"2021-01-15T18:43:19.534961+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4226,"ondisk_log_size":4226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34566393,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4109","reported_seq":"4169","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.523821+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:19.523821+0000","last_peered":"2021-01-15T18:43:19.523821+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:19.523821+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4109,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33628326,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4192","reported_seq":"4257","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.208237+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:24.208237+0000","last_peered":"2021-01-15T18:43:24.208237+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:24.208237+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4192,"ondisk_log_size":4192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34300070,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4214","reported_seq":"7107","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.168755+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:24.168755+0000","last_peered":"2021-01-15T18:43:24.168755+0000","last_clean":"2021-01-15T18:43:24.168755+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:24.168755+0000","last_undegraded":"2021-01-15T18:43:24.168755+0000","last_fullsized":"2021-01-15T18:43:24.168755+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4214,"ondisk_log_size":4214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34463910,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4377","reported_seq":"4444","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.129185+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:24.129185+0000","last_peered":"2021-01-15T18:43:24.129185+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:24.129185+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4377,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35807398,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":34980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4354","reported_seq":"4416","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:19.440670+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:19.440670+0000","last_peered":"2021-01-15T18:43:19.440670+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:19.440670+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4354,"ondisk_log_size":4354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35586380,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":34771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4222","reported_seq":"4750","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:23.331108+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:23.331108+0000","last_peered":"2021-01-15T18:43:23.331108+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:23.331108+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4222,"ondisk_log_size":4222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34562214,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":33754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3060}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":550144276,"num_objects":67170,"num_object_clones":0,"num_object_copies":268680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67197,"num_write_kb":537366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6069,"num_bytes_recovered":49422336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040056320,"data_stored":1040056320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67224,"ondisk_log_size":67224,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404492,"kb_used_data":267012,"kb_used_omap":0,"kb_used_meta":137471,"kb_avail":93967348,"statfs":{"total":96636764160,"available":96222564352,"internally_reserved":0,"allocated":273420288,"data_stored":273215675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":140770356},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,12,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":3.8900000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.0979999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":5.9619999999999997},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":5.9039999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.0510000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.032}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409984,"kb_used_data":270708,"kb_used_omap":0,"kb_used_meta":139264,"kb_avail":93961856,"statfs":{"total":96636764160,"available":96216940544,"internally_reserved":0,"allocated":277204992,"data_stored":277000379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":142606336},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":6.8220000000000001},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":4.1159999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":1.7869999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":2.5760000000000001},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":8.8040000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280562,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403032,"kb_used_data":271652,"kb_used_omap":0,"kb_used_meta":131391,"kb_avail":93968808,"statfs":{"total":96636764160,"available":96224059392,"internally_reserved":0,"allocated":278171648,"data_stored":277967035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":134544436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":17.309000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":2.2389999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.347},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":2.6400000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319128,"kb_used_data":205580,"kb_used_omap":0,"kb_used_meta":113535,"kb_avail":94052712,"statfs":{"total":96636764160,"available":96309977088,"internally_reserved":0,"allocated":210513920,"data_stored":210309307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":116259892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":6.9569999999999999},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":2.4969999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":6.6849999999999996},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":6.6509999999999998},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":2.6000000000000001}]}]}]}} 2021-01-15T18:43:26.208 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:27.086 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 207 20 68785 68765 2.59427 2.11719 0.0601279 0.0602128 2021-01-15T18:43:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:27.532 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15336/272204 objects degraded (5.634%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68051,"data_bytes":557361428,"bytes_used":1590075392,"bytes_avail":384956981248,"bytes_total":386547056640,"degraded_objects":15336,"degraded_total":272204,"degraded_ratio":0.056340097867775635,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":95501,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":2497352,"read_op_per_sec":0,"write_op_per_sec":304},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:27.592 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:27.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:28.086 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 208 20 69052 69032 2.59183 2.08594 0.0532214 0.0602649 2021-01-15T18:43:28.672 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:28.789 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2021-01-15T18:43:28.452362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":559409428,"num_objects":68301,"num_object_clones":0,"num_object_copies":273204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68328,"num_write_kb":546414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6134,"num_bytes_recovered":49954816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68355,"ondisk_log_size":68355,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1548560,"kb_used_data":1031816,"kb_used_omap":2,"kb_used_meta":516797,"kb_avail":375938800,"statfs":{"total":386547056640,"available":384961331200,"internally_reserved":0,"allocated":1056579584,"data_stored":1055761132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":529200284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,0,14,16,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":107,"apply_latency_ms":107,"commit_latency_ns":107000000,"apply_latency_ns":107000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24723456,"num_objects":3018,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":24144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":1679360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009110"},"pg_stats":[{"pgid":"1.f","version":"38'4150","reported_seq":"4198","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.417096+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:24.417096+0000","last_peered":"2021-01-15T18:43:24.417096+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:24.417096+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4150,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33996800,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2796}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4160","reported_seq":"7080","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.552143+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:24.552143+0000","last_peered":"2021-01-15T18:43:24.552143+0000","last_clean":"2021-01-15T18:43:24.552143+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:24.552143+0000","last_undegraded":"2021-01-15T18:43:24.552143+0000","last_fullsized":"2021-01-15T18:43:24.552143+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4160,"ondisk_log_size":4160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34037926,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":33252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4262","reported_seq":"4343","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:28.060800+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:28.060800+0000","last_peered":"2021-01-15T18:43:28.060800+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:28.060800+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4262,"ondisk_log_size":4262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34885715,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4261,"num_write_kb":34074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4317","reported_seq":"4365","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.514344+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:24.514344+0000","last_peered":"2021-01-15T18:43:24.514344+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:24.514344+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4317,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35364864,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":34536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4329","reported_seq":"4919","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.771673+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:24.771673+0000","last_peered":"2021-01-15T18:43:24.771673+0000","last_clean":"2021-01-15T18:43:24.771673+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:24.771673+0000","last_undegraded":"2021-01-15T18:43:24.771673+0000","last_fullsized":"2021-01-15T18:43:24.771673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4329,"ondisk_log_size":4329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35414182,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":34596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4189","reported_seq":"4398","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.741040+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:24.741040+0000","last_peered":"2021-01-15T18:43:24.741040+0000","last_clean":"2021-01-15T18:43:24.741040+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:24.741040+0000","last_undegraded":"2021-01-15T18:43:24.741040+0000","last_fullsized":"2021-01-15T18:43:24.741040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4189,"ondisk_log_size":4189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34283686,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4187,"num_write_kb":33487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4219","reported_seq":"4275","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.494111+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:24.494111+0000","last_peered":"2021-01-15T18:43:24.494111+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:24.494111+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4219,"ondisk_log_size":4219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34521254,"num_objects":4215,"num_object_clones":0,"num_object_copies":16860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":33724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2823}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4393","reported_seq":"4469","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:28.303730+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:28.303730+0000","last_peered":"2021-01-15T18:43:28.303730+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:28.303730+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4393,"ondisk_log_size":4393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35975251,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4303","reported_seq":"4346","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.579085+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:24.579085+0000","last_peered":"2021-01-15T18:43:24.579085+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:24.579085+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4303,"ondisk_log_size":4303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35237971,"num_objects":4302,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":34413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2866}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4308","reported_seq":"7246","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.495307+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:24.495307+0000","last_peered":"2021-01-15T18:43:24.495307+0000","last_clean":"2021-01-15T18:43:24.495307+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:24.495307+0000","last_undegraded":"2021-01-15T18:43:24.495307+0000","last_fullsized":"2021-01-15T18:43:24.495307+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4308,"ondisk_log_size":4308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35238137,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":34420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4194","reported_seq":"4254","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.494201+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:24.494201+0000","last_peered":"2021-01-15T18:43:24.494201+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:24.494201+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4194,"ondisk_log_size":4194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34324646,"num_objects":4191,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":33527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4192","reported_seq":"4257","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.208237+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:24.208237+0000","last_peered":"2021-01-15T18:43:24.208237+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:24.208237+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4192,"ondisk_log_size":4192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34300070,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2817}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4214","reported_seq":"7107","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:24.168755+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:24.168755+0000","last_peered":"2021-01-15T18:43:24.168755+0000","last_clean":"2021-01-15T18:43:24.168755+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:24.168755+0000","last_undegraded":"2021-01-15T18:43:24.168755+0000","last_fullsized":"2021-01-15T18:43:24.168755+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4214,"ondisk_log_size":4214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34463910,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4377","reported_seq":"4444","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.129185+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:24.129185+0000","last_peered":"2021-01-15T18:43:24.129185+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:24.129185+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4377,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35807398,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":34980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4446","reported_seq":"4508","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:24.558923+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:24.558923+0000","last_peered":"2021-01-15T18:43:24.558923+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:24.558923+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4446,"ondisk_log_size":4446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36340044,"num_objects":4438,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":35507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2952}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4302","reported_seq":"4960","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:28.360264+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:28.360264+0000","last_peered":"2021-01-15T18:43:28.360264+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:28.360264+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4302,"ondisk_log_size":4302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35217574,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":34394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":2351104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":559409428,"num_objects":68301,"num_object_clones":0,"num_object_copies":273204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68328,"num_write_kb":546414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6134,"num_bytes_recovered":49954816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057157120,"data_stored":1057157120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68355,"ondisk_log_size":68355,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404104,"kb_used_data":272452,"kb_used_omap":0,"kb_used_meta":131711,"kb_avail":93967736,"statfs":{"total":96636764160,"available":96222961664,"internally_reserved":0,"allocated":278990848,"data_stored":278786235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":134872116},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":2.206},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":2.319}]},{"osd":1,"last update":"Fri Jan 15 18:42:34 2021","interfaces":[{"interface":"back","average":{"1min":2.8500000000000001,"5min":3.0270000000000001,"15min":3.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":11.448,"5min":11.448,"15min":11.448},"last":3.0190000000000001},{"interface":"front","average":{"1min":2.8799999999999999,"5min":2.1280000000000001,"15min":2.0019999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":3.1070000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":7.0910000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":8.3729999999999993}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405736,"kb_used_data":276320,"kb_used_omap":0,"kb_used_meta":129408,"kb_avail":93966104,"statfs":{"total":96636764160,"available":96221290496,"internally_reserved":0,"allocated":282951680,"data_stored":282747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":132513792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,5,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.2429999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":2.3530000000000002,"15min":2.3079999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.926,"5min":13.926,"15min":13.926},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.9750000000000001,"15min":2.0369999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.685,"5min":10.685,"15min":10.685},"last":0.98599999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":3.6640000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280562,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403032,"kb_used_data":271652,"kb_used_omap":0,"kb_used_meta":131391,"kb_avail":93968808,"statfs":{"total":96636764160,"available":96224059392,"internally_reserved":0,"allocated":278171648,"data_stored":277967035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":134544436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:42:32 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.6299999999999999,"15min":2.8359999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.085,"5min":13.311,"15min":13.311},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.8559999999999999,"5min":2.4449999999999998,"15min":2.21},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":12.925000000000001,"5min":17.337,"15min":17.337},"last":17.309000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":2.2389999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.347},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":2.6400000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335688,"kb_used_data":211392,"kb_used_omap":0,"kb_used_meta":124287,"kb_avail":94036152,"statfs":{"total":96636764160,"available":96293019648,"internally_reserved":0,"allocated":216465408,"data_stored":216260795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":127269940},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":5.7009999999999996,"5min":3.2970000000000002,"15min":2.8969999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":35.850999999999999,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.556,"5min":1.992,"15min":1.8979999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":32.216999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.75900000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":2.633,"15min":2.694},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":9.5519999999999996,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.117,"5min":1.5449999999999999,"15min":1.617},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.6269999999999998,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.6230000000000002,"5min":3.8780000000000001,"15min":4.0869999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.975999999999999,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.4359999999999999,"15min":2.5670000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.915,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.877}]}]}]}} 2021-01-15T18:43:29.091 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 209 20 69318 69298 2.58931 2.07812 0.0373508 0.0603289 2021-01-15T18:43:30.092 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 210 20 69600 69580 2.58746 2.20312 0.0290721 0.0603593 2021-01-15T18:43:31.093 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 211 20 69831 69811 2.58374 1.80469 0.0637095 0.0604603 2021-01-15T18:43:31.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:32.093 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 212 20 70110 70090 2.58183 2.17969 0.0708838 0.0605005 2021-01-15T18:43:33.096 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 213 20 70395 70375 2.58013 2.22656 0.0653741 0.0605451 2021-01-15T18:43:34.096 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 214 20 70675 70655 2.5783 2.1875 0.0633645 0.0605871 2021-01-15T18:43:35.096 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 215 20 70892 70872 2.57419 1.69531 0.0543933 0.0606841 2021-01-15T18:43:36.096 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 216 20 71178 71158 2.57262 2.23438 0.0777792 0.060721 2021-01-15T18:43:37.097 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 217 20 71456 71436 2.57077 2.17188 0.068492 0.0607615 2021-01-15T18:43:38.099 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 218 20 71733 71713 2.56888 2.16406 0.0764604 0.060811 2021-01-15T18:43:39.101 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 219 20 72009 71989 2.56696 2.15625 0.0470566 0.0608556 2021-01-15T18:43:40.101 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:2021-01-15T18:43:40.101849+0000 min lat: 0.00287162 max lat: 15.9249 avg lat: 0.0608997 2021-01-15T18:43:40.102 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:43:40.102 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 220 20 72288 72268 2.5652 2.17969 0.0558507 0.0608997 2021-01-15T18:43:41.103 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 221 20 72529 72509 2.5621 1.88281 0.0784998 0.0609706 2021-01-15T18:43:41.565 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:41.581 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:41.664 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2021-01-15T18:43:40.458647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590604564,"num_objects":72109,"num_object_clones":0,"num_object_copies":288436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72136,"num_write_kb":576878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":51224576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72163,"ondisk_log_size":72163,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1639604,"kb_used_data":1092120,"kb_used_omap":2,"kb_used_meta":547453,"kb_avail":375847756,"statfs":{"total":386547056640,"available":384868102144,"internally_reserved":0,"allocated":1118330880,"data_stored":1117512428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":560592028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,12,9,12,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":103,"apply_latency_ms":103,"commit_latency_ns":103000000,"apply_latency_ns":103000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31195136,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":30464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006285"},"pg_stats":[{"pgid":"1.f","version":"38'4403","reported_seq":"4448","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.624905+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:39.624905+0000","last_peered":"2021-01-15T18:43:39.624905+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:39.624905+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4403,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36069376,"num_objects":4403,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":35224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4397","reported_seq":"7319","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.743359+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:39.743359+0000","last_peered":"2021-01-15T18:43:39.743359+0000","last_clean":"2021-01-15T18:43:39.743359+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:39.743359+0000","last_undegraded":"2021-01-15T18:43:39.743359+0000","last_fullsized":"2021-01-15T18:43:39.743359+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4397,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35979430,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":35148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4426","reported_seq":"4507","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:38.238409+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:38.238409+0000","last_peered":"2021-01-15T18:43:38.238409+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:38.238409+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4426,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36229203,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":35386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4580","reported_seq":"4628","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.605595+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:39.605595+0000","last_peered":"2021-01-15T18:43:39.605595+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:39.605595+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4580,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37519360,"num_objects":4580,"num_object_clones":0,"num_object_copies":18320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4580,"num_write_kb":36640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4576","reported_seq":"5166","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.557142+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:39.557142+0000","last_peered":"2021-01-15T18:43:39.557142+0000","last_clean":"2021-01-15T18:43:39.557142+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:39.557142+0000","last_undegraded":"2021-01-15T18:43:39.557142+0000","last_fullsized":"2021-01-15T18:43:39.557142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4576,"ondisk_log_size":4576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37437606,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":36572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4469","reported_seq":"4677","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.801202+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:39.801202+0000","last_peered":"2021-01-15T18:43:39.801202+0000","last_clean":"2021-01-15T18:43:39.801202+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:39.801202+0000","last_undegraded":"2021-01-15T18:43:39.801202+0000","last_fullsized":"2021-01-15T18:43:39.801202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4469,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36577446,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4485","reported_seq":"4541","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.543509+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:39.543509+0000","last_peered":"2021-01-15T18:43:39.543509+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:39.543509+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4485,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36700326,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4573","reported_seq":"4649","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:38.370962+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:38.370962+0000","last_peered":"2021-01-15T18:43:38.370962+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:38.370962+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4573,"ondisk_log_size":4573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37449811,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4572,"num_write_kb":36573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3120}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4559","reported_seq":"4603","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.465526+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:39.465526+0000","last_peered":"2021-01-15T18:43:39.465526+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:39.465526+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4559,"ondisk_log_size":4559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37335123,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4558,"num_write_kb":36461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4560","reported_seq":"7498","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.586160+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:39.586160+0000","last_peered":"2021-01-15T18:43:39.586160+0000","last_clean":"2021-01-15T18:43:39.586160+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:39.586160+0000","last_undegraded":"2021-01-15T18:43:39.586160+0000","last_fullsized":"2021-01-15T18:43:39.586160+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4560,"ondisk_log_size":4560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37302521,"num_objects":4555,"num_object_clones":0,"num_object_copies":18220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4557,"num_write_kb":36436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4455","reported_seq":"4514","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.602337+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:39.602337+0000","last_peered":"2021-01-15T18:43:39.602337+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:39.602337+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4455,"ondisk_log_size":4455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36462758,"num_objects":4452,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":35615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3071}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4468","reported_seq":"4534","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.137700+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:39.137700+0000","last_peered":"2021-01-15T18:43:39.137700+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:39.137700+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4468,"ondisk_log_size":4468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36561062,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":35711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4456","reported_seq":"7349","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.134045+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:39.134045+0000","last_peered":"2021-01-15T18:43:39.134045+0000","last_clean":"2021-01-15T18:43:39.134045+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:39.134045+0000","last_undegraded":"2021-01-15T18:43:39.134045+0000","last_fullsized":"2021-01-15T18:43:39.134045+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4456,"ondisk_log_size":4456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36446374,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4454,"num_write_kb":35604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4599","reported_seq":"4666","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.230549+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:39.230549+0000","last_peered":"2021-01-15T18:43:39.230549+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:39.230549+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4599,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37626022,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4597,"num_write_kb":36756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4706","reported_seq":"4768","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.455172+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:39.455172+0000","last_peered":"2021-01-15T18:43:39.455172+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:39.455172+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4706,"ondisk_log_size":4706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38469964,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4703,"num_write_kb":37587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4451","reported_seq":"5420","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:38.370248+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:38.370248+0000","last_peered":"2021-01-15T18:43:38.370248+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:38.370248+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4451,"ondisk_log_size":4451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36438182,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4449,"num_write_kb":35586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":3620864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590604564,"num_objects":72109,"num_object_clones":0,"num_object_copies":288436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72136,"num_write_kb":576878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":51224576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118875648,"data_stored":1118875648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72163,"ondisk_log_size":72163,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431576,"kb_used_data":288660,"kb_used_omap":0,"kb_used_meta":142911,"kb_avail":93940264,"statfs":{"total":96636764160,"available":96194830336,"internally_reserved":0,"allocated":295587840,"data_stored":295383227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146340916},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,8,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.4510000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.552}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":3.7629999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":3.8300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.6139999999999999},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.6499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421200,"kb_used_data":287048,"kb_used_omap":0,"kb_used_meta":134144,"kb_avail":93950640,"statfs":{"total":96636764160,"available":96205455360,"internally_reserved":0,"allocated":293937152,"data_stored":293732539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":137363456},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":5.8929999999999998},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.2309999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":20.885000000000002},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":20.98}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":6.2069999999999999},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":3.3410000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434676,"kb_used_data":287916,"kb_used_omap":0,"kb_used_meta":146751,"kb_avail":93937164,"statfs":{"total":96636764160,"available":96191655936,"internally_reserved":0,"allocated":294825984,"data_stored":294621371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":150273076},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.1259999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":2.8050000000000002},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.2350000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.4080000000000004},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":4.649}]}]},{"osd":1,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352152,"kb_used_data":228496,"kb_used_omap":0,"kb_used_meta":123647,"kb_avail":94019688,"statfs":{"total":96636764160,"available":96276160512,"internally_reserved":0,"allocated":233979904,"data_stored":233775291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":126614580},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":13.656000000000001},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.2960000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.82999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.54},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.72599999999999998}]}]}]}} 2021-01-15T18:43:41.666 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:42.103 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 222 20 72795 72775 2.55992 2.07812 0.0820402 0.0610178 2021-01-15T18:43:43.105 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 223 20 73063 73043 2.5578 2.09375 0.0982716 0.0610737 2021-01-15T18:43:43.401 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:43.402 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15116/288436 objects degraded (5.241%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72109,"data_bytes":590604564,"bytes_used":1678954496,"bytes_avail":384868102144,"bytes_total":386547056640,"degraded_objects":15116,"degraded_total":288436,"degraded_ratio":0.052406773079643323,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105758,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":1843608,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:43.455 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:43.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:44.105 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 224 20 73284 73264 2.55409 1.72656 0.0900631 0.0611582 2021-01-15T18:43:44.484 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:44.536 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2021-01-15T18:43:44.460966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594274580,"num_objects":72557,"num_object_clones":0,"num_object_copies":290228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72584,"num_write_kb":580462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6376,"num_bytes_recovered":51937280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72611,"ondisk_log_size":72611,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1654788,"kb_used_data":1102696,"kb_used_omap":2,"kb_used_meta":552061,"kb_avail":375832572,"statfs":{"total":386547056640,"available":384852553728,"internally_reserved":0,"allocated":1129160704,"data_stored":1128342252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":565310620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,6,12,20,19,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":107,"apply_latency_ms":107,"commit_latency_ns":107000000,"apply_latency_ns":107000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25804800,"num_objects":3150,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3150,"num_write_kb":25200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006091"},"pg_stats":[{"pgid":"1.f","version":"38'4403","reported_seq":"4448","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.624905+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:39.624905+0000","last_peered":"2021-01-15T18:43:39.624905+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:39.624905+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4403,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36069376,"num_objects":4403,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":35224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4397","reported_seq":"7319","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.743359+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:39.743359+0000","last_peered":"2021-01-15T18:43:39.743359+0000","last_clean":"2021-01-15T18:43:39.743359+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:39.743359+0000","last_undegraded":"2021-01-15T18:43:39.743359+0000","last_fullsized":"2021-01-15T18:43:39.743359+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4397,"ondisk_log_size":4397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35979430,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4395,"num_write_kb":35148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4490","reported_seq":"4570","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:43.164089+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:43.164089+0000","last_peered":"2021-01-15T18:43:43.164089+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:43.164089+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4490,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36753491,"num_objects":4487,"num_object_clones":0,"num_object_copies":17948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4489,"num_write_kb":35898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4580","reported_seq":"4628","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.605595+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:39.605595+0000","last_peered":"2021-01-15T18:43:39.605595+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:39.605595+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4580,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37519360,"num_objects":4580,"num_object_clones":0,"num_object_copies":18320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4580,"num_write_kb":36640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4576","reported_seq":"5166","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.557142+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:39.557142+0000","last_peered":"2021-01-15T18:43:39.557142+0000","last_clean":"2021-01-15T18:43:39.557142+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:39.557142+0000","last_undegraded":"2021-01-15T18:43:39.557142+0000","last_fullsized":"2021-01-15T18:43:39.557142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4576,"ondisk_log_size":4576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37437606,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":36572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4469","reported_seq":"4677","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.801202+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:39.801202+0000","last_peered":"2021-01-15T18:43:39.801202+0000","last_clean":"2021-01-15T18:43:39.801202+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:39.801202+0000","last_undegraded":"2021-01-15T18:43:39.801202+0000","last_fullsized":"2021-01-15T18:43:39.801202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4469,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36577446,"num_objects":4466,"num_object_clones":0,"num_object_copies":17864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4485","reported_seq":"4541","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.543509+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:39.543509+0000","last_peered":"2021-01-15T18:43:39.543509+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:39.543509+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4485,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36700326,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3089}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4647","reported_seq":"4723","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:43.155102+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:43.155102+0000","last_peered":"2021-01-15T18:43:43.155102+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:43.155102+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4647,"ondisk_log_size":4647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38056019,"num_objects":4646,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4646,"num_write_kb":37165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4559","reported_seq":"4603","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.465526+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:39.465526+0000","last_peered":"2021-01-15T18:43:39.465526+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:39.465526+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4559,"ondisk_log_size":4559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37335123,"num_objects":4558,"num_object_clones":0,"num_object_copies":18232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4558,"num_write_kb":36461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4560","reported_seq":"7498","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:39.586160+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:39.586160+0000","last_peered":"2021-01-15T18:43:39.586160+0000","last_clean":"2021-01-15T18:43:39.586160+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:39.586160+0000","last_undegraded":"2021-01-15T18:43:39.586160+0000","last_fullsized":"2021-01-15T18:43:39.586160+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4560,"ondisk_log_size":4560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37302521,"num_objects":4555,"num_object_clones":0,"num_object_copies":18220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4557,"num_write_kb":36436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4455","reported_seq":"4514","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.602337+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:39.602337+0000","last_peered":"2021-01-15T18:43:39.602337+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:39.602337+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4455,"ondisk_log_size":4455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36462758,"num_objects":4452,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":35615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3071}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4541","reported_seq":"4607","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.137069+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:44.137069+0000","last_peered":"2021-01-15T18:43:44.137069+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:44.137069+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4541,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37159078,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":36295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4533","reported_seq":"7426","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.239743+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:44.239743+0000","last_peered":"2021-01-15T18:43:44.239743+0000","last_clean":"2021-01-15T18:43:44.239743+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:44.239743+0000","last_undegraded":"2021-01-15T18:43:44.239743+0000","last_fullsized":"2021-01-15T18:43:44.239743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4533,"ondisk_log_size":4533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37077158,"num_objects":4527,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":36220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4680","reported_seq":"4747","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.178183+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:44.178183+0000","last_peered":"2021-01-15T18:43:44.178183+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:44.178183+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4680,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38289574,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":37404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4706","reported_seq":"4768","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:39.455172+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:39.455172+0000","last_peered":"2021-01-15T18:43:39.455172+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:39.455172+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4706,"ondisk_log_size":4706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38469964,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4703,"num_write_kb":37587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4530","reported_seq":"5670","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:43.420799+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:43.420799+0000","last_peered":"2021-01-15T18:43:43.420799+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:43.420799+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4530,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37085350,"num_objects":4528,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":36218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3675}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594274580,"num_objects":72557,"num_object_clones":0,"num_object_copies":290228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72584,"num_write_kb":580462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6376,"num_bytes_recovered":51937280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1129422848,"data_stored":1129422848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72611,"ondisk_log_size":72611,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215154,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431576,"kb_used_data":288660,"kb_used_omap":0,"kb_used_meta":142911,"kb_avail":93940264,"statfs":{"total":96636764160,"available":96194830336,"internally_reserved":0,"allocated":295587840,"data_stored":295383227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146340916},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,8,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.4510000000000001},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.552}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":3.7629999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":3.8300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.6139999999999999},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.6499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437000,"kb_used_data":292352,"kb_used_omap":0,"kb_used_meta":144640,"kb_avail":93934840,"statfs":{"total":96636764160,"available":96189276160,"internally_reserved":0,"allocated":299368448,"data_stored":299163835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":148111360},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.6299999999999999},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.98099999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":3.1600000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.282},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":8.4740000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434060,"kb_used_data":293188,"kb_used_omap":0,"kb_used_meta":140863,"kb_avail":93937780,"statfs":{"total":96636764160,"available":96192286720,"internally_reserved":0,"allocated":300224512,"data_stored":300019899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":144243764},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":0.54500000000000004},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.754}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":1.3759999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.0529999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.258},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":5.6989999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352152,"kb_used_data":228496,"kb_used_omap":0,"kb_used_meta":123647,"kb_avail":94019688,"statfs":{"total":96636764160,"available":96276160512,"internally_reserved":0,"allocated":233979904,"data_stored":233775291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":126614580},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":13.656000000000001},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.2960000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.82999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.54},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.72599999999999998}]}]}]}} 2021-01-15T18:43:45.105 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 225 20 73556 73536 2.55219 2.125 0.0922366 0.0612076 2021-01-15T18:43:46.106 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 226 20 73836 73816 2.55057 2.1875 0.0564115 0.0612489 2021-01-15T18:43:47.106 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 227 20 74108 74088 2.54869 2.125 0.0474019 0.0612918 2021-01-15T18:43:47.539 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:48.109 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 228 20 74373 74353 2.54656 2.07031 0.0590471 0.0613442 2021-01-15T18:43:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:48.959 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:48.983 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2021-01-15T18:43:48.465904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603179284,"num_objects":73644,"num_object_clones":0,"num_object_copies":294576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73671,"num_write_kb":589158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":52568064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73698,"ondisk_log_size":73698,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1677532,"kb_used_data":1118532,"kb_used_omap":2,"kb_used_meta":558973,"kb_avail":375809828,"statfs":{"total":386547056640,"available":384829263872,"internally_reserved":0,"allocated":1145376768,"data_stored":1144558316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":572388508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,7,10,23,32,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":110,"apply_latency_ms":110,"commit_latency_ns":110000000,"apply_latency_ns":110000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23617536,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":1867776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008279"},"pg_stats":[{"pgid":"1.f","version":"38'4493","reported_seq":"4540","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.607097+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:44.607097+0000","last_peered":"2021-01-15T18:43:44.607097+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:44.607097+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4493,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36806656,"num_objects":4493,"num_object_clones":0,"num_object_copies":17972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":35944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4489","reported_seq":"7411","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.716740+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:44.716740+0000","last_peered":"2021-01-15T18:43:44.716740+0000","last_clean":"2021-01-15T18:43:44.716740+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:44.716740+0000","last_undegraded":"2021-01-15T18:43:44.716740+0000","last_fullsized":"2021-01-15T18:43:44.716740+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4489,"ondisk_log_size":4489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36733094,"num_objects":4485,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4487,"num_write_kb":35884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4572","reported_seq":"4653","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.307890+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:48.307890+0000","last_peered":"2021-01-15T18:43:48.307890+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:48.307890+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4572,"ondisk_log_size":4572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37425235,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4571,"num_write_kb":36554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4651","reported_seq":"4699","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.576328+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:44.576328+0000","last_peered":"2021-01-15T18:43:44.576328+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:44.576328+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4651,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":37208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4679","reported_seq":"5269","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.725966+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:44.725966+0000","last_peered":"2021-01-15T18:43:44.725966+0000","last_clean":"2021-01-15T18:43:44.725966+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:44.725966+0000","last_undegraded":"2021-01-15T18:43:44.725966+0000","last_fullsized":"2021-01-15T18:43:44.725966+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4679,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38281382,"num_objects":4674,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4539","reported_seq":"4748","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.717025+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:44.717025+0000","last_peered":"2021-01-15T18:43:44.717025+0000","last_clean":"2021-01-15T18:43:44.717025+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:44.717025+0000","last_undegraded":"2021-01-15T18:43:44.717025+0000","last_fullsized":"2021-01-15T18:43:44.717025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4539,"ondisk_log_size":4539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37150886,"num_objects":4536,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4537,"num_write_kb":36287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4574","reported_seq":"4630","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.557674+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:44.557674+0000","last_peered":"2021-01-15T18:43:44.557674+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:44.557674+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4574,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37429414,"num_objects":4570,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4572,"num_write_kb":36564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4737","reported_seq":"4813","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.373863+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:48.373863+0000","last_peered":"2021-01-15T18:43:48.373863+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:48.373863+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4737,"ondisk_log_size":4737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38793299,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4630","reported_seq":"4674","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.559609+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:44.559609+0000","last_peered":"2021-01-15T18:43:44.559609+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:44.559609+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4630,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37916755,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":37029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4630","reported_seq":"7569","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.568884+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:44.568884+0000","last_peered":"2021-01-15T18:43:44.568884+0000","last_clean":"2021-01-15T18:43:44.568884+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:44.568884+0000","last_undegraded":"2021-01-15T18:43:44.568884+0000","last_fullsized":"2021-01-15T18:43:44.568884+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4630,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37875961,"num_objects":4625,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4627,"num_write_kb":36996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4532","reported_seq":"4591","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.517808+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:44.517808+0000","last_peered":"2021-01-15T18:43:44.517808+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:44.517808+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4532,"ondisk_log_size":4532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37093542,"num_objects":4529,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":36231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3148}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4541","reported_seq":"4607","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.137069+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:44.137069+0000","last_peered":"2021-01-15T18:43:44.137069+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:44.137069+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4541,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37159078,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":36295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4533","reported_seq":"7426","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:44.239743+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:44.239743+0000","last_peered":"2021-01-15T18:43:44.239743+0000","last_clean":"2021-01-15T18:43:44.239743+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:44.239743+0000","last_undegraded":"2021-01-15T18:43:44.239743+0000","last_fullsized":"2021-01-15T18:43:44.239743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4533,"ondisk_log_size":4533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37077158,"num_objects":4527,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":36220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4680","reported_seq":"4747","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.178183+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:44.178183+0000","last_peered":"2021-01-15T18:43:44.178183+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:44.178183+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4680,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38289574,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":37404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3233}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4797","reported_seq":"4859","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:44.575350+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:44.575350+0000","last_peered":"2021-01-15T18:43:44.575350+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:44.575350+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4797,"ondisk_log_size":4797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39215436,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":38315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4621","reported_seq":"5917","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.377287+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:48.377287+0000","last_peered":"2021-01-15T18:43:48.377287+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:48.377287+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4621,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37830822,"num_objects":4619,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":603179284,"num_objects":73644,"num_object_clones":0,"num_object_copies":294576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73671,"num_write_kb":589158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":52568064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1145749504,"data_stored":1145749504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73698,"ondisk_log_size":73698,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443060,"kb_used_data":293804,"kb_used_omap":0,"kb_used_meta":149247,"kb_avail":93928780,"statfs":{"total":96636764160,"available":96183070720,"internally_reserved":0,"allocated":300855296,"data_stored":300650683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":152828980},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,4,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":0.68500000000000005},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":2.6139999999999999},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.6499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436344,"kb_used_data":297592,"kb_used_omap":0,"kb_used_meta":138752,"kb_avail":93935496,"statfs":{"total":96636764160,"available":96189947904,"internally_reserved":0,"allocated":304734208,"data_stored":304529595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":142082048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,6,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":9.2119999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":7.2699999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":7.7220000000000004},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":7.6079999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":9.3230000000000004},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":8.032}]}]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434060,"kb_used_data":293188,"kb_used_omap":0,"kb_used_meta":140863,"kb_avail":93937780,"statfs":{"total":96636764160,"available":96192286720,"internally_reserved":0,"allocated":300224512,"data_stored":300019899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":144243764},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":0.54500000000000004},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.754}]},{"osd":2,"last update":"Fri Jan 15 18:42:47 2021","interfaces":[{"interface":"back","average":{"1min":3.3180000000000001,"5min":2.8319999999999999,"15min":2.4239999999999999},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":15.74,"5min":27.77,"15min":27.77},"last":1.3759999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.0619999999999998,"15min":2.5680000000000001},"min":{"1min":0.58399999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":11.099,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.0529999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.258},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":5.6989999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364068,"kb_used_data":233948,"kb_used_omap":0,"kb_used_meta":130111,"kb_avail":94007772,"statfs":{"total":96636764160,"available":96263958528,"internally_reserved":0,"allocated":239562752,"data_stored":239358139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":133233716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,6,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.1470000000000002},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":21.791},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.76200000000000001}]}]}]}} 2021-01-15T18:43:48.985 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:49.110 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 229 20 74633 74613 2.5443 2.03125 0.0703025 0.0613977 2021-01-15T18:43:50.110 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 230 20 74879 74859 2.54159 1.92188 0.0606154 0.0614632 2021-01-15T18:43:50.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:50.283 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14952/294576 objects degraded (5.076%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":280,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73644,"data_bytes":603179284,"bytes_used":1717792768,"bytes_avail":384829263872,"bytes_total":386547056640,"degraded_objects":14952,"degraded_total":294576,"degraded_ratio":0.050757699201564282,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":155540,"recovering_keys_per_sec":0,"num_objects_recovered":228,"num_bytes_recovered":1867776,"num_keys_recovered":0,"write_bytes_sec":1966771,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:50.339 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:50.339 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:51.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 231 20 75139 75119 2.53936 2.03125 0.0838995 0.061517 2021-01-15T18:43:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:51.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2021-01-15T18:43:50.466418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":611944724,"num_objects":74714,"num_object_clones":0,"num_object_copies":298856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74741,"num_write_kb":597718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":52568064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74768,"ondisk_log_size":74768,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1692988,"kb_used_data":1134964,"kb_used_omap":2,"kb_used_meta":558013,"kb_avail":375794372,"statfs":{"total":386547056640,"available":384813436928,"internally_reserved":0,"allocated":1162203136,"data_stored":1161384684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":571405468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,6,2,12,19,25,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":113,"apply_latency_ms":113,"commit_latency_ns":113000000,"apply_latency_ns":113000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30318592,"num_objects":3701,"num_object_clones":0,"num_object_copies":14804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":29608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008280"},"pg_stats":[{"pgid":"1.f","version":"38'4581","reported_seq":"4629","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.550733+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:49.550733+0000","last_peered":"2021-01-15T18:43:49.550733+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:49.550733+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4581,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4578","reported_seq":"7500","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.779447+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:49.779447+0000","last_peered":"2021-01-15T18:43:49.779447+0000","last_clean":"2021-01-15T18:43:49.779447+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:49.779447+0000","last_undegraded":"2021-01-15T18:43:49.779447+0000","last_fullsized":"2021-01-15T18:43:49.779447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4578,"ondisk_log_size":4578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37462182,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":36596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4572","reported_seq":"4653","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.307890+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:48.307890+0000","last_peered":"2021-01-15T18:43:48.307890+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:48.307890+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4572,"ondisk_log_size":4572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37425235,"num_objects":4569,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4571,"num_write_kb":36554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3153}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4726","reported_seq":"4774","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.312340+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:49.312340+0000","last_peered":"2021-01-15T18:43:49.312340+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:49.312340+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4726,"ondisk_log_size":4726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38715392,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4726,"num_write_kb":37808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4756","reported_seq":"5346","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.823925+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:49.823925+0000","last_peered":"2021-01-15T18:43:49.823925+0000","last_clean":"2021-01-15T18:43:49.823925+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:49.823925+0000","last_undegraded":"2021-01-15T18:43:49.823925+0000","last_fullsized":"2021-01-15T18:43:49.823925+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4756,"ondisk_log_size":4756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38912166,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":38012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4617","reported_seq":"4826","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.588187+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:49.588187+0000","last_peered":"2021-01-15T18:43:49.588187+0000","last_clean":"2021-01-15T18:43:49.588187+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:49.588187+0000","last_undegraded":"2021-01-15T18:43:49.588187+0000","last_fullsized":"2021-01-15T18:43:49.588187+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4617,"ondisk_log_size":4617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37789862,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4668","reported_seq":"4724","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.537233+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:49.537233+0000","last_peered":"2021-01-15T18:43:49.537233+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:49.537233+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4668,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38199462,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4737","reported_seq":"4813","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.373863+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:48.373863+0000","last_peered":"2021-01-15T18:43:48.373863+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:48.373863+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4737,"ondisk_log_size":4737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38793299,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3284}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4703","reported_seq":"4747","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.514433+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:49.514433+0000","last_peered":"2021-01-15T18:43:49.514433+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:49.514433+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4703,"ondisk_log_size":4703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38514771,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":37613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4725","reported_seq":"7664","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.520220+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:49.520220+0000","last_peered":"2021-01-15T18:43:49.520220+0000","last_clean":"2021-01-15T18:43:49.520220+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:49.520220+0000","last_undegraded":"2021-01-15T18:43:49.520220+0000","last_fullsized":"2021-01-15T18:43:49.520220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4725,"ondisk_log_size":4725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38654201,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4619","reported_seq":"4679","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.518502+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:49.518502+0000","last_peered":"2021-01-15T18:43:49.518502+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:49.518502+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4619,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37806246,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4624","reported_seq":"4690","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.189947+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:49.189947+0000","last_peered":"2021-01-15T18:43:49.189947+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:49.189947+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4624,"ondisk_log_size":4624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37839014,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4622,"num_write_kb":36959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4597","reported_seq":"7490","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.225202+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:49.225202+0000","last_peered":"2021-01-15T18:43:49.225202+0000","last_clean":"2021-01-15T18:43:49.225202+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:49.225202+0000","last_undegraded":"2021-01-15T18:43:49.225202+0000","last_fullsized":"2021-01-15T18:43:49.225202+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4597,"ondisk_log_size":4597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37601446,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4595,"num_write_kb":36732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4767","reported_seq":"4831","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.222752+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:49.222752+0000","last_peered":"2021-01-15T18:43:49.222752+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:49.222752+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4767,"ondisk_log_size":4767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39002278,"num_objects":4762,"num_object_clones":0,"num_object_copies":19048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":38100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4877","reported_seq":"4938","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.684304+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:49.684304+0000","last_peered":"2021-01-15T18:43:49.684304+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:49.684304+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4877,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39870796,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4874,"num_write_kb":38955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3383}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4621","reported_seq":"5917","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:48.377287+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:48.377287+0000","last_peered":"2021-01-15T18:43:48.377287+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:48.377287+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4621,"ondisk_log_size":4621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37830822,"num_objects":4619,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":36946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":4964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":611944724,"num_objects":74714,"num_object_clones":0,"num_object_copies":298856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74741,"num_write_kb":597718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":52568064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1162493952,"data_stored":1162493952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74768,"ondisk_log_size":74768,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442600,"kb_used_data":299168,"kb_used_omap":0,"kb_used_meta":143423,"kb_avail":93929240,"statfs":{"total":96636764160,"available":96183541760,"internally_reserved":0,"allocated":306348032,"data_stored":306143419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146865204},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,4,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.498},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.93300000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":1.073},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":1.978}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":6.6920000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.2160000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":436344,"kb_used_data":297592,"kb_used_omap":0,"kb_used_meta":138752,"kb_avail":93935496,"statfs":{"total":96636764160,"available":96189947904,"internally_reserved":0,"allocated":304734208,"data_stored":304529595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":142082048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,4,6,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":3.6179999999999999,"15min":3.7090000000000001},"min":{"1min":0.38400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":16.326000000000001,"5min":36.380000000000003,"15min":36.380000000000003},"last":9.2119999999999997},{"interface":"front","average":{"1min":2.351,"5min":1.677,"15min":1.216},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":17.352,"5min":21.344999999999999,"15min":21.344999999999999},"last":7.2699999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":7.7220000000000004},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":7.6079999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":9.3230000000000004},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":8.032}]}]},{"osd":0,"up_from":22,"seq":94489280567,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449848,"kb_used_data":298496,"kb_used_omap":0,"kb_used_meta":151359,"kb_avail":93921992,"statfs":{"total":96636764160,"available":96176119808,"internally_reserved":0,"allocated":305659904,"data_stored":305455291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":154991668},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,4,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.8359999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":7.3929999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.9249999999999998},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":2.0070000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":0.995}]}]},{"osd":1,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364196,"kb_used_data":239708,"kb_used_omap":0,"kb_used_meta":124479,"kb_avail":94007644,"statfs":{"total":96636764160,"available":96263827456,"internally_reserved":0,"allocated":245460992,"data_stored":245256379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":127466548},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.3079999999999998},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":2.6269999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.5419999999999998},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":1.585}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.492},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.581}]}]}]}} 2021-01-15T18:43:52.113 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 232 20 75393 75373 2.53697 1.98438 0.0428842 0.0615733 2021-01-15T18:43:53.113 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 233 20 75646 75626 2.53456 1.97656 0.0627372 0.0616359 2021-01-15T18:43:54.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 234 20 75903 75883 2.5323 2.00781 0.050644 0.0616895 2021-01-15T18:43:54.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:55.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 235 20 76168 76148 2.53034 2.07031 0.0877077 0.0617339 2021-01-15T18:43:55.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:55.679 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:55.697 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2021-01-15T18:43:54.470284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615459092,"num_objects":75143,"num_object_clones":0,"num_object_copies":300572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75170,"num_write_kb":601150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":53280768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75197,"ondisk_log_size":75197,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1707516,"kb_used_data":1145072,"kb_used_omap":2,"kb_used_meta":562429,"kb_avail":375779844,"statfs":{"total":386547056640,"available":384798560256,"internally_reserved":0,"allocated":1172553728,"data_stored":1171735276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":575927452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,8,2,5,22,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24854528,"num_objects":3034,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":24272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2056192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009784"},"pg_stats":[{"pgid":"1.f","version":"38'4581","reported_seq":"4629","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.550733+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:49.550733+0000","last_peered":"2021-01-15T18:43:49.550733+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:49.550733+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4581,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4578","reported_seq":"7500","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.779447+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:49.779447+0000","last_peered":"2021-01-15T18:43:49.779447+0000","last_clean":"2021-01-15T18:43:49.779447+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:49.779447+0000","last_undegraded":"2021-01-15T18:43:49.779447+0000","last_fullsized":"2021-01-15T18:43:49.779447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4578,"ondisk_log_size":4578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37462182,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":36596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4644","reported_seq":"4724","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.278878+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:53.278878+0000","last_peered":"2021-01-15T18:43:53.278878+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:53.278878+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4644,"ondisk_log_size":4644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38015059,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":37130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4726","reported_seq":"4774","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.312340+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:49.312340+0000","last_peered":"2021-01-15T18:43:49.312340+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:49.312340+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4726,"ondisk_log_size":4726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38715392,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4726,"num_write_kb":37808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4756","reported_seq":"5346","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.823925+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:49.823925+0000","last_peered":"2021-01-15T18:43:49.823925+0000","last_clean":"2021-01-15T18:43:49.823925+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:49.823925+0000","last_undegraded":"2021-01-15T18:43:49.823925+0000","last_fullsized":"2021-01-15T18:43:49.823925+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4756,"ondisk_log_size":4756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38912166,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":38012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4617","reported_seq":"4826","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.588187+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:49.588187+0000","last_peered":"2021-01-15T18:43:49.588187+0000","last_clean":"2021-01-15T18:43:49.588187+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:49.588187+0000","last_undegraded":"2021-01-15T18:43:49.588187+0000","last_fullsized":"2021-01-15T18:43:49.588187+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4617,"ondisk_log_size":4617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37789862,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4668","reported_seq":"4724","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.537233+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:49.537233+0000","last_peered":"2021-01-15T18:43:49.537233+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:49.537233+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4668,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38199462,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4819","reported_seq":"4895","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.332178+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:53.332178+0000","last_peered":"2021-01-15T18:43:53.332178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:53.332178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4819,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39465043,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4703","reported_seq":"4747","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.514433+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:49.514433+0000","last_peered":"2021-01-15T18:43:49.514433+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:49.514433+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4703,"ondisk_log_size":4703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38514771,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":37613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4725","reported_seq":"7664","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:49.520220+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:49.520220+0000","last_peered":"2021-01-15T18:43:49.520220+0000","last_clean":"2021-01-15T18:43:49.520220+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:49.520220+0000","last_undegraded":"2021-01-15T18:43:49.520220+0000","last_fullsized":"2021-01-15T18:43:49.520220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4725,"ondisk_log_size":4725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38654201,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4619","reported_seq":"4679","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.518502+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:49.518502+0000","last_peered":"2021-01-15T18:43:49.518502+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:49.518502+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4619,"ondisk_log_size":4619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37806246,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4617,"num_write_kb":36927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4697","reported_seq":"4762","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.265698+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:54.265698+0000","last_peered":"2021-01-15T18:43:54.265698+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:54.265698+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4697,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38437030,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4666","reported_seq":"7559","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.264125+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:54.264125+0000","last_peered":"2021-01-15T18:43:54.264125+0000","last_clean":"2021-01-15T18:43:54.264125+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:54.264125+0000","last_undegraded":"2021-01-15T18:43:54.264125+0000","last_fullsized":"2021-01-15T18:43:54.264125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4666,"ondisk_log_size":4666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38166694,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4842","reported_seq":"4909","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.180113+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:54.180113+0000","last_peered":"2021-01-15T18:43:54.180113+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:54.180113+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4842,"ondisk_log_size":4842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39616678,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4840,"num_write_kb":38700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4877","reported_seq":"4938","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:49.684304+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:49.684304+0000","last_peered":"2021-01-15T18:43:49.684304+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:49.684304+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4877,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39870796,"num_objects":4869,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4874,"num_write_kb":38955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3383}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4679","reported_seq":"6149","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.416033+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:53.416033+0000","last_peered":"2021-01-15T18:43:53.416033+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:53.416033+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4679,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38305958,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":695,"num_bytes_recovered":5677056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3988}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":615459092,"num_objects":75143,"num_object_clones":0,"num_object_copies":300572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75170,"num_write_kb":601150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":53280768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1172905984,"data_stored":1172905984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75197,"ondisk_log_size":75197,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215156,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442600,"kb_used_data":299168,"kb_used_omap":0,"kb_used_meta":143423,"kb_avail":93929240,"statfs":{"total":96636764160,"available":96183541760,"internally_reserved":0,"allocated":306348032,"data_stored":306143419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146865204},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,4,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.498},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.93300000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":1.073},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":1.978}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":6.6920000000000002},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.2160000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451908,"kb_used_data":302728,"kb_used_omap":0,"kb_used_meta":149184,"kb_avail":93919932,"statfs":{"total":96636764160,"available":96174010368,"internally_reserved":0,"allocated":309993472,"data_stored":309788859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":152764416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.4630000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":1.4370000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448812,"kb_used_data":303468,"kb_used_omap":0,"kb_used_meta":145343,"kb_avail":93923028,"statfs":{"total":96636764160,"available":96177180672,"internally_reserved":0,"allocated":310751232,"data_stored":310546619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":148831284},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.3490000000000002},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":2.548}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.2450000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.7609999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":17.059999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364196,"kb_used_data":239708,"kb_used_omap":0,"kb_used_meta":124479,"kb_avail":94007644,"statfs":{"total":96636764160,"available":96263827456,"internally_reserved":0,"allocated":245460992,"data_stored":245256379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":127466548},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.3079999999999998},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":2.6269999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.5419999999999998},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":1.585}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.492},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.581}]}]}]}} 2021-01-15T18:43:55.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:43:56.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 236 20 76388 76368 2.5269 1.71875 0.0586883 0.0618132 2021-01-15T18:43:56.690 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:56.691 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14865/304012 objects degraded (4.890%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":287,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76003,"data_bytes":622504212,"bytes_used":1780711424,"bytes_avail":384766345216,"bytes_total":386547056640,"degraded_objects":14865,"degraded_total":304012,"degraded_ratio":0.048896096206728676,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":111865,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"write_bytes_sec":2350544,"read_op_per_sec":0,"write_op_per_sec":286},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:43:56.728 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:43:56.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:43:57.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 237 20 76609 76589 2.52352 1.72656 0.0300506 0.0618885 2021-01-15T18:43:57.663 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:43:57.667 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:43:57.680 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2021-01-15T18:43:56.470791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622504212,"num_objects":76003,"num_object_clones":0,"num_object_copies":304012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76030,"num_write_kb":608030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":53280768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76057,"ondisk_log_size":76057,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1738976,"kb_used_data":1155556,"kb_used_omap":2,"kb_used_meta":583421,"kb_avail":375748384,"statfs":{"total":386547056640,"available":384766345216,"internally_reserved":0,"allocated":1183289344,"data_stored":1182470892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":597423260},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,12,30,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":106,"apply_latency_ms":106,"commit_latency_ns":106000000,"apply_latency_ns":106000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28229632,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009824"},"pg_stats":[{"pgid":"1.f","version":"38'4664","reported_seq":"4712","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.640712+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:54.640712+0000","last_peered":"2021-01-15T18:43:54.640712+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:54.640712+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4664,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38207488,"num_objects":4664,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4662","reported_seq":"7583","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.717117+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:54.717117+0000","last_peered":"2021-01-15T18:43:54.717117+0000","last_clean":"2021-01-15T18:43:54.717117+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:54.717117+0000","last_undegraded":"2021-01-15T18:43:54.717117+0000","last_fullsized":"2021-01-15T18:43:54.717117+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4662,"ondisk_log_size":4662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38150310,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4660,"num_write_kb":37268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4644","reported_seq":"4724","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.278878+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:53.278878+0000","last_peered":"2021-01-15T18:43:53.278878+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:53.278878+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4644,"ondisk_log_size":4644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38015059,"num_objects":4641,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":37130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3225}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4806","reported_seq":"4854","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.509097+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:54.509097+0000","last_peered":"2021-01-15T18:43:54.509097+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:54.509097+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4806,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39370752,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3374}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4837","reported_seq":"5427","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.820526+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:54.820526+0000","last_peered":"2021-01-15T18:43:54.820526+0000","last_clean":"2021-01-15T18:43:54.820526+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:54.820526+0000","last_undegraded":"2021-01-15T18:43:54.820526+0000","last_fullsized":"2021-01-15T18:43:54.820526+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4837,"ondisk_log_size":4837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39575718,"num_objects":4832,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":38660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4690","reported_seq":"4899","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.435115+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:54.435115+0000","last_peered":"2021-01-15T18:43:54.435115+0000","last_clean":"2021-01-15T18:43:54.435115+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:54.435115+0000","last_undegraded":"2021-01-15T18:43:54.435115+0000","last_fullsized":"2021-01-15T18:43:54.435115+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4690,"ondisk_log_size":4690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38387878,"num_objects":4687,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4688,"num_write_kb":37495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4782","reported_seq":"4838","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.651858+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:54.651858+0000","last_peered":"2021-01-15T18:43:54.651858+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:54.651858+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4782,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39133350,"num_objects":4778,"num_object_clones":0,"num_object_copies":19112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3386}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4819","reported_seq":"4895","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.332178+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:53.332178+0000","last_peered":"2021-01-15T18:43:53.332178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:53.332178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4819,"ondisk_log_size":4819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39465043,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4793","reported_seq":"4836","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.638619+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:54.638619+0000","last_peered":"2021-01-15T18:43:54.638619+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:54.638619+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4793,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39252051,"num_objects":4792,"num_object_clones":0,"num_object_copies":19168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":38333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4812","reported_seq":"7751","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.610648+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:54.610648+0000","last_peered":"2021-01-15T18:43:54.610648+0000","last_clean":"2021-01-15T18:43:54.610648+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:54.610648+0000","last_undegraded":"2021-01-15T18:43:54.610648+0000","last_fullsized":"2021-01-15T18:43:54.610648+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4812,"ondisk_log_size":4812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39366905,"num_objects":4807,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4707","reported_seq":"4766","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.671640+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:54.671640+0000","last_peered":"2021-01-15T18:43:54.671640+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:54.671640+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4707,"ondisk_log_size":4707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38527142,"num_objects":4704,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":37631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4697","reported_seq":"4762","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.265698+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:54.265698+0000","last_peered":"2021-01-15T18:43:54.265698+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:54.265698+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4697,"ondisk_log_size":4697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38437030,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4666","reported_seq":"7559","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:54.264125+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:54.264125+0000","last_peered":"2021-01-15T18:43:54.264125+0000","last_clean":"2021-01-15T18:43:54.264125+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:54.264125+0000","last_undegraded":"2021-01-15T18:43:54.264125+0000","last_fullsized":"2021-01-15T18:43:54.264125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4666,"ondisk_log_size":4666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38166694,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":37284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4842","reported_seq":"4909","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.180113+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:54.180113+0000","last_peered":"2021-01-15T18:43:54.180113+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:54.180113+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4842,"ondisk_log_size":4842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39616678,"num_objects":4837,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4840,"num_write_kb":38700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'4957","reported_seq":"5018","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:54.679284+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:54.679284+0000","last_peered":"2021-01-15T18:43:54.679284+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:54.679284+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4957,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40526156,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":39595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4679","reported_seq":"6149","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:53.416033+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:53.416033+0000","last_peered":"2021-01-15T18:43:53.416033+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:53.416033+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4679,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38305958,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":695,"num_bytes_recovered":5677056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3988}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622504212,"num_objects":76003,"num_object_clones":0,"num_object_copies":304012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76030,"num_write_kb":608030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":53280768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1183858688,"data_stored":1183858688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76057,"ondisk_log_size":76057,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458172,"kb_used_data":304320,"kb_used_omap":0,"kb_used_meta":153855,"kb_avail":93913668,"statfs":{"total":96636764160,"available":96167596032,"internally_reserved":0,"allocated":311623680,"data_stored":311419067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":157547572},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,16],"upper_bound":64},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":2.653},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":2.3140000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":2.3999999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":1.1870000000000001},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":7.6689999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451908,"kb_used_data":302728,"kb_used_omap":0,"kb_used_meta":149184,"kb_avail":93919932,"statfs":{"total":96636764160,"available":96174010368,"internally_reserved":0,"allocated":309993472,"data_stored":309788859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":152764416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.4630000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:42:52 2021","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":2.5019999999999998,"15min":2.3599999999999999},"min":{"1min":0.70699999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.917000000000002,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":4.1639999999999997,"5min":2.6269999999999998,"15min":2.3900000000000001},"min":{"1min":0.58999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.901,"5min":13.901,"15min":13.901},"last":1.4370000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448812,"kb_used_data":303468,"kb_used_omap":0,"kb_used_meta":145343,"kb_avail":93923028,"statfs":{"total":96636764160,"available":96177180672,"internally_reserved":0,"allocated":310751232,"data_stored":310546619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":148831284},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.3490000000000002},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":2.548}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.2450000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.7609999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:42:56 2021","interfaces":[{"interface":"back","average":{"1min":4.6760000000000002,"5min":4.0549999999999997,"15min":4.4809999999999999},"min":{"1min":0.51600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":16.978999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":7.1189999999999998,"5min":3.2999999999999998,"15min":2.52},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":30.213000000000001,"5min":30.213000000000001,"15min":30.213000000000001},"last":17.059999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380084,"kb_used_data":245040,"kb_used_omap":0,"kb_used_meta":135039,"kb_avail":93991756,"statfs":{"total":96636764160,"available":96247558144,"internally_reserved":0,"allocated":250920960,"data_stored":250716347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":138279988},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":13.212999999999999},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":10.470000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.75},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.57899999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":2.5270000000000001},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.742}]}]}]}} 2021-01-15T18:43:58.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 238 20 76838 76818 2.52043 1.78906 0.109925 0.0619811 2021-01-15T18:43:59.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout: 239 20 77101 77081 2.51849 2.05469 0.0646699 0.0620269 2021-01-15T18:44:00.114 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Total time run: 240.045 2021-01-15T18:44:00.115 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Total writes made: 77320 2021-01-15T18:44:00.115 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Write size: 8192 2021-01-15T18:44:00.115 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Object size: 8192 2021-01-15T18:44:00.115 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Bandwidth (MB/sec): 2.51646 2021-01-15T18:44:00.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Stddev Bandwidth: 1.59511 2021-01-15T18:44:00.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Max bandwidth (MB/sec): 8.53125 2021-01-15T18:44:00.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Min bandwidth (MB/sec): 0 2021-01-15T18:44:00.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Average IOPS: 322 2021-01-15T18:44:00.116 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Stddev IOPS: 204.187 2021-01-15T18:44:00.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Max IOPS: 1092 2021-01-15T18:44:00.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Min IOPS: 0 2021-01-15T18:44:00.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Average Latency(s): 0.0620821 2021-01-15T18:44:00.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Stddev Latency(s): 0.300833 2021-01-15T18:44:00.117 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Max latency(s): 15.9249 2021-01-15T18:44:00.118 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Min latency(s): 0.00287162 2021-01-15T18:44:00.130 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Cleaning up (deleting benchmark objects) 2021-01-15T18:44:00.683 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:01.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:01.366 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:01.406 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2021-01-15T18:44:00.473832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632219924,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77216,"num_write_kb":617518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6633,"num_bytes_recovered":54042624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77243,"ondisk_log_size":77243,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1750736,"kb_used_data":1175048,"kb_used_omap":2,"kb_used_meta":575677,"kb_avail":375736624,"statfs":{"total":386547056640,"available":384754302976,"internally_reserved":0,"allocated":1203249152,"data_stored":1202430700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":589493404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,8,9,13,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":111,"apply_latency_ms":111,"commit_latency_ns":111000000,"apply_latency_ns":111000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29040640,"num_objects":3545,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":28360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007927"},"pg_stats":[{"pgid":"1.f","version":"38'4745","reported_seq":"4792","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.500772+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:59.500772+0000","last_peered":"2021-01-15T18:43:59.500772+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:59.500772+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4745,"ondisk_log_size":4745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38871040,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":37960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4732","reported_seq":"7654","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.777182+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:59.777182+0000","last_peered":"2021-01-15T18:43:59.777182+0000","last_clean":"2021-01-15T18:43:59.777182+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:59.777182+0000","last_undegraded":"2021-01-15T18:43:59.777182+0000","last_fullsized":"2021-01-15T18:43:59.777182+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4732,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38723750,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4724","reported_seq":"4804","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.365811+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:58.365811+0000","last_peered":"2021-01-15T18:43:58.365811+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:58.365811+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4724,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38670419,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":37770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4890","reported_seq":"4938","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.615425+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:59.615425+0000","last_peered":"2021-01-15T18:43:59.615425+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:59.615425+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4890,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40058880,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4890,"num_write_kb":39120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4912","reported_seq":"5502","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.809174+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:59.809174+0000","last_peered":"2021-01-15T18:43:59.809174+0000","last_clean":"2021-01-15T18:43:59.809174+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:59.809174+0000","last_undegraded":"2021-01-15T18:43:59.809174+0000","last_fullsized":"2021-01-15T18:43:59.809174+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4912,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40190118,"num_objects":4907,"num_object_clones":0,"num_object_copies":19628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4761","reported_seq":"4970","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.680208+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:59.680208+0000","last_peered":"2021-01-15T18:43:59.680208+0000","last_clean":"2021-01-15T18:43:59.680208+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:59.680208+0000","last_undegraded":"2021-01-15T18:43:59.680208+0000","last_fullsized":"2021-01-15T18:43:59.680208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4761,"ondisk_log_size":4761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38969510,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":38063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4860","reported_seq":"4915","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.600691+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:59.600691+0000","last_peered":"2021-01-15T18:43:59.600691+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:59.600691+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4860,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39772326,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4901","reported_seq":"4977","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.412926+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:58.412926+0000","last_peered":"2021-01-15T18:43:58.412926+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:58.412926+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4901,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40136787,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4900,"num_write_kb":39197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3448}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4857","reported_seq":"4901","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.577372+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:59.577372+0000","last_peered":"2021-01-15T18:43:59.577372+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:59.577372+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4857,"ondisk_log_size":4857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39776339,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":38845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4879","reported_seq":"7818","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.602766+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:59.602766+0000","last_peered":"2021-01-15T18:43:59.602766+0000","last_clean":"2021-01-15T18:43:59.602766+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:59.602766+0000","last_undegraded":"2021-01-15T18:43:59.602766+0000","last_fullsized":"2021-01-15T18:43:59.602766+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4879,"ondisk_log_size":4879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39915769,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4876,"num_write_kb":38988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4774","reported_seq":"4834","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.609338+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:59.609338+0000","last_peered":"2021-01-15T18:43:59.609338+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:59.609338+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4774,"ondisk_log_size":4774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39076006,"num_objects":4771,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":38167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4779","reported_seq":"4844","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.253243+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:59.253243+0000","last_peered":"2021-01-15T18:43:59.253243+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:59.253243+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4779,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39108774,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4735","reported_seq":"7627","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.241732+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:59.241732+0000","last_peered":"2021-01-15T18:43:59.241732+0000","last_clean":"2021-01-15T18:43:59.241732+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:59.241732+0000","last_undegraded":"2021-01-15T18:43:59.241732+0000","last_fullsized":"2021-01-15T18:43:59.241732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4735,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38731942,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4933","reported_seq":"4999","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.127155+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:59.127155+0000","last_peered":"2021-01-15T18:43:59.127155+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:59.127155+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4933,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40362150,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5023","reported_seq":"5085","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.600659+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:59.600659+0000","last_peered":"2021-01-15T18:43:59.600659+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:59.600659+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5023,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41066828,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5020,"num_write_kb":40123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4738","reported_seq":"6394","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.330786+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:58.330786+0000","last_peered":"2021-01-15T18:43:58.330786+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:58.330786+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4738,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38789286,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":788,"num_bytes_recovered":6438912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":632219924,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77216,"num_write_kb":617518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6633,"num_bytes_recovered":54042624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203679232,"data_stored":1203679232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77243,"ondisk_log_size":77243,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456896,"kb_used_data":309124,"kb_used_omap":0,"kb_used_meta":147775,"kb_avail":93914944,"statfs":{"total":96636764160,"available":96168902656,"internally_reserved":0,"allocated":316542976,"data_stored":316338363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":151321652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,2,6,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.4420000000000002},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.548}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":4.6630000000000003},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.7850000000000001},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":14.436}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450564,"kb_used_data":307464,"kb_used_omap":0,"kb_used_meta":143104,"kb_avail":93921276,"statfs":{"total":96636764160,"available":96175386624,"internally_reserved":0,"allocated":314843136,"data_stored":314638523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":146538496},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":18.93},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.1840000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.319}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.2250000000000001},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":3.2930000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463924,"kb_used_data":308268,"kb_used_omap":0,"kb_used_meta":155647,"kb_avail":93907916,"statfs":{"total":96636764160,"available":96161705984,"internally_reserved":0,"allocated":315666432,"data_stored":315461819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":159382580},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":5.0759999999999996},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":3.0270000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":4.9630000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.8410000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.8370000000000002},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":2.948}]}]},{"osd":1,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379352,"kb_used_data":250192,"kb_used_omap":0,"kb_used_meta":129151,"kb_avail":93992488,"statfs":{"total":96636764160,"available":96248307712,"internally_reserved":0,"allocated":256196608,"data_stored":255991995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":132250676},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.661},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.72499999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.1820000000000004},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":4.202}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.427},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.90600000000000003}]}]}]}} 2021-01-15T18:44:01.408 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:02.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:02.708 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14772/308756 objects degraded (4.784%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":293,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77189,"data_bytes":632219924,"bytes_used":1792753664,"bytes_avail":384754302976,"bytes_total":386547056640,"degraded_objects":14772,"degraded_total":308756,"degraded_ratio":0.047843604658694888,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":122797,"recovering_keys_per_sec":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"write_bytes_sec":1688461,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:44:02.765 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:02.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:03.457 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:03.521 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2021-01-15T18:44:02.474509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632219924,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77216,"num_write_kb":617518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6633,"num_bytes_recovered":54042624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77243,"ondisk_log_size":77243,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1750736,"kb_used_data":1175048,"kb_used_omap":2,"kb_used_meta":575677,"kb_avail":375736624,"statfs":{"total":386547056640,"available":384754302976,"internally_reserved":0,"allocated":1203249152,"data_stored":1202430700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":589493404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,8,9,13,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":111,"apply_latency_ms":111,"commit_latency_ns":111000000,"apply_latency_ns":111000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20275200,"num_objects":2475,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":19800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008091"},"pg_stats":[{"pgid":"1.f","version":"38'4745","reported_seq":"4792","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.500772+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:43:59.500772+0000","last_peered":"2021-01-15T18:43:59.500772+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:43:59.500772+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4745,"ondisk_log_size":4745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38871040,"num_objects":4745,"num_object_clones":0,"num_object_copies":18980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":37960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4732","reported_seq":"7654","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.777182+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:43:59.777182+0000","last_peered":"2021-01-15T18:43:59.777182+0000","last_clean":"2021-01-15T18:43:59.777182+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:43:59.777182+0000","last_undegraded":"2021-01-15T18:43:59.777182+0000","last_fullsized":"2021-01-15T18:43:59.777182+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4732,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38723750,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4724","reported_seq":"4804","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.365811+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:43:58.365811+0000","last_peered":"2021-01-15T18:43:58.365811+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:43:58.365811+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4724,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38670419,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":37770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4890","reported_seq":"4938","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.615425+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:43:59.615425+0000","last_peered":"2021-01-15T18:43:59.615425+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:43:59.615425+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4890,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40058880,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4890,"num_write_kb":39120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4912","reported_seq":"5502","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.809174+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:43:59.809174+0000","last_peered":"2021-01-15T18:43:59.809174+0000","last_clean":"2021-01-15T18:43:59.809174+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:43:59.809174+0000","last_undegraded":"2021-01-15T18:43:59.809174+0000","last_fullsized":"2021-01-15T18:43:59.809174+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4912,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40190118,"num_objects":4907,"num_object_clones":0,"num_object_copies":19628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4761","reported_seq":"4970","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.680208+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:43:59.680208+0000","last_peered":"2021-01-15T18:43:59.680208+0000","last_clean":"2021-01-15T18:43:59.680208+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:43:59.680208+0000","last_undegraded":"2021-01-15T18:43:59.680208+0000","last_fullsized":"2021-01-15T18:43:59.680208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4761,"ondisk_log_size":4761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38969510,"num_objects":4758,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":38063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4860","reported_seq":"4915","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.600691+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:43:59.600691+0000","last_peered":"2021-01-15T18:43:59.600691+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:43:59.600691+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4860,"ondisk_log_size":4860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39772326,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":38852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4901","reported_seq":"4977","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.412926+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:43:58.412926+0000","last_peered":"2021-01-15T18:43:58.412926+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:43:58.412926+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4901,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40136787,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4900,"num_write_kb":39197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3448}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4857","reported_seq":"4901","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.577372+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:43:59.577372+0000","last_peered":"2021-01-15T18:43:59.577372+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:43:59.577372+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4857,"ondisk_log_size":4857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39776339,"num_objects":4856,"num_object_clones":0,"num_object_copies":19424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":38845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4879","reported_seq":"7818","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.602766+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:43:59.602766+0000","last_peered":"2021-01-15T18:43:59.602766+0000","last_clean":"2021-01-15T18:43:59.602766+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:43:59.602766+0000","last_undegraded":"2021-01-15T18:43:59.602766+0000","last_fullsized":"2021-01-15T18:43:59.602766+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4879,"ondisk_log_size":4879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39915769,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4876,"num_write_kb":38988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4774","reported_seq":"4834","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.609338+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:43:59.609338+0000","last_peered":"2021-01-15T18:43:59.609338+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:43:59.609338+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4774,"ondisk_log_size":4774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39076006,"num_objects":4771,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":38167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4779","reported_seq":"4844","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.253243+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:43:59.253243+0000","last_peered":"2021-01-15T18:43:59.253243+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:43:59.253243+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4779,"ondisk_log_size":4779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39108774,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4735","reported_seq":"7627","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:43:59.241732+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:43:59.241732+0000","last_peered":"2021-01-15T18:43:59.241732+0000","last_clean":"2021-01-15T18:43:59.241732+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:43:59.241732+0000","last_undegraded":"2021-01-15T18:43:59.241732+0000","last_fullsized":"2021-01-15T18:43:59.241732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4735,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38731942,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'4933","reported_seq":"4999","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.127155+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:43:59.127155+0000","last_peered":"2021-01-15T18:43:59.127155+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:43:59.127155+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4933,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40362150,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5023","reported_seq":"5085","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:59.600659+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:43:59.600659+0000","last_peered":"2021-01-15T18:43:59.600659+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:43:59.600659+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5023,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41066828,"num_objects":5015,"num_object_clones":0,"num_object_copies":20060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5020,"num_write_kb":40123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4738","reported_seq":"6394","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:43:58.330786+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:43:58.330786+0000","last_peered":"2021-01-15T18:43:58.330786+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:43:58.330786+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4738,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38789286,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":37882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":788,"num_bytes_recovered":6438912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":632219924,"num_objects":77189,"num_object_clones":0,"num_object_copies":308756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77216,"num_write_kb":617518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6633,"num_bytes_recovered":54042624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1203679232,"data_stored":1203679232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77243,"ondisk_log_size":77243,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456896,"kb_used_data":309124,"kb_used_omap":0,"kb_used_meta":147775,"kb_avail":93914944,"statfs":{"total":96636764160,"available":96168902656,"internally_reserved":0,"allocated":316542976,"data_stored":316338363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":151321652},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,2,6,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":3.3199999999999998,"15min":3.1120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":16.355,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.4420000000000002},{"interface":"front","average":{"1min":4.7069999999999999,"5min":3.3839999999999999,"15min":3.4390000000000001},"min":{"1min":0.59899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":16.629000000000001,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.548}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":4.6630000000000003},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:42:55 2021","interfaces":[{"interface":"back","average":{"1min":3.5209999999999999,"5min":2.6070000000000002,"15min":2.1989999999999998},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":15.164,"5min":15.164,"15min":15.164},"last":4.7850000000000001},{"interface":"front","average":{"1min":3.528,"5min":3.0339999999999998,"15min":2.9089999999999998},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":13.662000000000001,"5min":18.300000000000001,"15min":18.300000000000001},"last":14.436}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450564,"kb_used_data":307464,"kb_used_omap":0,"kb_used_meta":143104,"kb_avail":93921276,"statfs":{"total":96636764160,"available":96175386624,"internally_reserved":0,"allocated":314843136,"data_stored":314638523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":146538496},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":18.93},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.1840000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.319}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.2250000000000001},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":3.2930000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463924,"kb_used_data":308268,"kb_used_omap":0,"kb_used_meta":155647,"kb_avail":93907916,"statfs":{"total":96636764160,"available":96161705984,"internally_reserved":0,"allocated":315666432,"data_stored":315461819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":159382580},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":5.0759999999999996},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":3.0270000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":4.9630000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.8410000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.8370000000000002},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":2.948}]}]},{"osd":1,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379352,"kb_used_data":250192,"kb_used_omap":0,"kb_used_meta":129151,"kb_avail":93992488,"statfs":{"total":96636764160,"available":96248307712,"internally_reserved":0,"allocated":256196608,"data_stored":255991995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":132250676},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.661},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.72499999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.1820000000000004},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":4.202}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.427},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.90600000000000003}]}]}]}} 2021-01-15T18:44:06.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:07.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:07.701 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:07.744 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2021-01-15T18:44:06.475515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625756460,"num_objects":76401,"num_object_clones":0,"num_object_copies":305604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14638,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":78324,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6767,"num_bytes_recovered":54968320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78351,"ondisk_log_size":78351,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1777940,"kb_used_data":1168868,"kb_used_omap":2,"kb_used_meta":608957,"kb_avail":375709420,"statfs":{"total":386547056640,"available":384726446080,"internally_reserved":0,"allocated":1196920832,"data_stored":1196102380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":623572124},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,9,20,14,4,4,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":97,"apply_latency_ms":97,"commit_latency_ns":97000000,"apply_latency_ns":97000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10297368,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3154,"num_write_kb":17641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":1687552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005231"},"pg_stats":[{"pgid":"1.f","version":"38'4816","reported_seq":"4868","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.383457+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:04.383457+0000","last_peered":"2021-01-15T18:44:04.383457+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:04.383457+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4816,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4816,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4804","reported_seq":"7726","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.485261+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:04.485261+0000","last_peered":"2021-01-15T18:44:04.485261+0000","last_clean":"2021-01-15T18:44:04.485261+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:04.485261+0000","last_undegraded":"2021-01-15T18:44:04.485261+0000","last_fullsized":"2021-01-15T18:44:04.485261+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4804,"ondisk_log_size":4804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38150310,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4802,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4799","reported_seq":"4884","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:03.349681+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:03.349681+0000","last_peered":"2021-01-15T18:44:03.349681+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:03.349681+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4799,"ondisk_log_size":4799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38441067,"num_objects":4694,"num_object_clones":0,"num_object_copies":18776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4798,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3280}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4940","reported_seq":"4990","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.447562+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:04.447562+0000","last_peered":"2021-01-15T18:44:04.447562+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:04.447562+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4940,"ondisk_log_size":4940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4993","reported_seq":"5583","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.569479+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:04.569479+0000","last_peered":"2021-01-15T18:44:04.569479+0000","last_clean":"2021-01-15T18:44:04.569479+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:04.569479+0000","last_undegraded":"2021-01-15T18:44:04.569479+0000","last_fullsized":"2021-01-15T18:44:04.569479+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4993,"ondisk_log_size":4993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39608486,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4991,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4814","reported_seq":"5023","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.425864+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:04.425864+0000","last_peered":"2021-01-15T18:44:04.425864+0000","last_clean":"2021-01-15T18:44:04.425864+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:04.425864+0000","last_undegraded":"2021-01-15T18:44:04.425864+0000","last_fullsized":"2021-01-15T18:44:04.425864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4814,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38600870,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4931","reported_seq":"4997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.442263+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:04.442263+0000","last_peered":"2021-01-15T18:44:04.442263+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:04.442263+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4931,"ondisk_log_size":4931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39272614,"num_objects":4795,"num_object_clones":0,"num_object_copies":19180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4929,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'4967","reported_seq":"5045","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:03.378477+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:03.378477+0000","last_peered":"2021-01-15T18:44:03.378477+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:03.378477+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4967,"ondisk_log_size":4967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40054867,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3439}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4924","reported_seq":"4974","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.431440+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:04.431440+0000","last_peered":"2021-01-15T18:44:04.431440+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:04.431440+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4924,"ondisk_log_size":4924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39358547,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3372}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4948","reported_seq":"7886","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.626815+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:04.626815+0000","last_peered":"2021-01-15T18:44:04.626815+0000","last_clean":"2021-01-15T18:44:04.626815+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:04.626815+0000","last_undegraded":"2021-01-15T18:44:04.626815+0000","last_fullsized":"2021-01-15T18:44:04.626815+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4948,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39399673,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4846","reported_seq":"4916","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.582970+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:04.582970+0000","last_peered":"2021-01-15T18:44:04.582970+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:04.582970+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4846,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38568102,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4849","reported_seq":"4915","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.140194+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:04.140194+0000","last_peered":"2021-01-15T18:44:04.140194+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:04.140194+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4849,"ondisk_log_size":4849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38650022,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4847,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4796","reported_seq":"7689","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.061503+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:04.061503+0000","last_peered":"2021-01-15T18:44:04.061503+0000","last_clean":"2021-01-15T18:44:04.061503+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:04.061503+0000","last_undegraded":"2021-01-15T18:44:04.061503+0000","last_fullsized":"2021-01-15T18:44:04.061503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4796,"ondisk_log_size":4796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38330534,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5005","reported_seq":"5072","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.139191+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:04.139191+0000","last_peered":"2021-01-15T18:44:04.139191+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:04.139191+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5005,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39985318,"num_objects":4882,"num_object_clones":0,"num_object_copies":19528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3440}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5102","reported_seq":"5168","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.549644+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:04.549644+0000","last_peered":"2021-01-15T18:44:04.549644+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:04.549644+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5102,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40468812,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4817","reported_seq":"6699","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:03.437536+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:03.437536+0000","last_peered":"2021-01-15T18:44:03.437536+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:03.437536+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4817,"ondisk_log_size":4817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38682790,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4815,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":901,"num_bytes_recovered":7364608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":625756460,"num_objects":76401,"num_object_clones":0,"num_object_copies":305604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14638,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":78324,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6767,"num_bytes_recovered":54968320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1194545152,"data_stored":1194545152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78351,"ondisk_log_size":78351,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463780,"kb_used_data":305532,"kb_used_omap":0,"kb_used_meta":158207,"kb_avail":93908060,"statfs":{"total":96636764160,"available":96161853440,"internally_reserved":0,"allocated":312864768,"data_stored":312660155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162004020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,12,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":9.5690000000000008},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":10.07}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":10.183999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":9.5120000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.831},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":9.6850000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464764,"kb_used_data":307188,"kb_used_omap":0,"kb_used_meta":157568,"kb_avail":93907076,"statfs":{"total":96636764160,"available":96160845824,"internally_reserved":0,"allocated":314560512,"data_stored":314355899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":161349632},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":18.93},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":3.1840000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.319}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.2250000000000001},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":3.2930000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456152,"kb_used_data":306468,"kb_used_omap":0,"kb_used_meta":149631,"kb_avail":93915688,"statfs":{"total":96636764160,"available":96169664512,"internally_reserved":0,"allocated":313823232,"data_stored":313618619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153222196},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.2010000000000001},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":14.942}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.0110000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":10.68},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":3.1760000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393244,"kb_used_data":249680,"kb_used_omap":0,"kb_used_meta":143551,"kb_avail":93978596,"statfs":{"total":96636764160,"available":96234082304,"internally_reserved":0,"allocated":255672320,"data_stored":255467707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146996276},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.069}]}]}]}} 2021-01-15T18:44:07.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:08.965 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:08.965 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14523/304764 objects degraded (4.765%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":299,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76191,"data_bytes":624036140,"bytes_used":1809838080,"bytes_avail":384737218560,"bytes_total":386547056640,"degraded_objects":14523,"degraded_total":304764,"degraded_ratio":0.047653266133795329,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":214863,"recovering_keys_per_sec":0,"num_objects_recovered":344,"num_bytes_recovered":2580480,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":917516,"read_op_per_sec":0,"write_op_per_sec":208},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2021-01-15T18:42:34.412097+0000","services":{}},"progress_events":{}} 2021-01-15T18:44:09.029 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:09.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:10.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:10.148 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:10.168 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2021-01-15T18:44:08.480668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624036140,"num_objects":76191,"num_object_clones":0,"num_object_copies":304764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":78534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6884,"num_bytes_recovered":55861248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78561,"ondisk_log_size":78561,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1767420,"kb_used_data":1164132,"kb_used_omap":2,"kb_used_meta":603133,"kb_avail":375719940,"statfs":{"total":386547056640,"available":384737218560,"internally_reserved":0,"allocated":1192071168,"data_stored":1191252716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":617608348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,10,18,13,12,3,0,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1531928,"num_objects":188,"num_object_clones":0,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2504,"num_write_kb":10761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":2580480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009877"},"pg_stats":[{"pgid":"1.f","version":"38'4816","reported_seq":"4868","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.383457+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:04.383457+0000","last_peered":"2021-01-15T18:44:04.383457+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:04.383457+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4816,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38436864,"num_objects":4692,"num_object_clones":0,"num_object_copies":18768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4816,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4804","reported_seq":"7726","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.485261+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:04.485261+0000","last_peered":"2021-01-15T18:44:04.485261+0000","last_clean":"2021-01-15T18:44:04.485261+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:04.485261+0000","last_undegraded":"2021-01-15T18:44:04.485261+0000","last_fullsized":"2021-01-15T18:44:04.485261+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4804,"ondisk_log_size":4804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38150310,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4802,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4873","reported_seq":"4965","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.303523+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:08.303523+0000","last_peered":"2021-01-15T18:44:08.303523+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:08.303523+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4873,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37834859,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4872,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'4940","reported_seq":"4990","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.447562+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:04.447562+0000","last_peered":"2021-01-15T18:44:04.447562+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:04.447562+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4940,"ondisk_log_size":4940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'4993","reported_seq":"5583","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.569479+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:04.569479+0000","last_peered":"2021-01-15T18:44:04.569479+0000","last_clean":"2021-01-15T18:44:04.569479+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:04.569479+0000","last_undegraded":"2021-01-15T18:44:04.569479+0000","last_fullsized":"2021-01-15T18:44:04.569479+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4993,"ondisk_log_size":4993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39608486,"num_objects":4836,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4991,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4814","reported_seq":"5023","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.425864+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:04.425864+0000","last_peered":"2021-01-15T18:44:04.425864+0000","last_clean":"2021-01-15T18:44:04.425864+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:04.425864+0000","last_undegraded":"2021-01-15T18:44:04.425864+0000","last_fullsized":"2021-01-15T18:44:04.425864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4814,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38600870,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4931","reported_seq":"4997","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.442263+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:04.442263+0000","last_peered":"2021-01-15T18:44:04.442263+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:04.442263+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4931,"ondisk_log_size":4931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39272614,"num_objects":4795,"num_object_clones":0,"num_object_copies":19180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4929,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3408}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5043","reported_seq":"5127","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.175014+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:08.175014+0000","last_peered":"2021-01-15T18:44:08.175014+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:08.175014+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5043,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39432275,"num_objects":4814,"num_object_clones":0,"num_object_copies":19256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4924","reported_seq":"4974","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.431440+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:04.431440+0000","last_peered":"2021-01-15T18:44:04.431440+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:04.431440+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4924,"ondisk_log_size":4924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39358547,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3372}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'4948","reported_seq":"7886","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.626815+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:04.626815+0000","last_peered":"2021-01-15T18:44:04.626815+0000","last_clean":"2021-01-15T18:44:04.626815+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:04.626815+0000","last_undegraded":"2021-01-15T18:44:04.626815+0000","last_fullsized":"2021-01-15T18:44:04.626815+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4948,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39399673,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4846","reported_seq":"4916","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.582970+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:04.582970+0000","last_peered":"2021-01-15T18:44:04.582970+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:04.582970+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4846,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38568102,"num_objects":4709,"num_object_clones":0,"num_object_copies":18836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4849","reported_seq":"4915","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.140194+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:04.140194+0000","last_peered":"2021-01-15T18:44:04.140194+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:04.140194+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4849,"ondisk_log_size":4849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38650022,"num_objects":4719,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4847,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4796","reported_seq":"7689","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:04.061503+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:04.061503+0000","last_peered":"2021-01-15T18:44:04.061503+0000","last_clean":"2021-01-15T18:44:04.061503+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:04.061503+0000","last_undegraded":"2021-01-15T18:44:04.061503+0000","last_fullsized":"2021-01-15T18:44:04.061503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4796,"ondisk_log_size":4796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38330534,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5005","reported_seq":"5072","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.139191+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:04.139191+0000","last_peered":"2021-01-15T18:44:04.139191+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:04.139191+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5005,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39985318,"num_objects":4882,"num_object_clones":0,"num_object_copies":19528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3440}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5102","reported_seq":"5168","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:04.549644+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:04.549644+0000","last_peered":"2021-01-15T18:44:04.549644+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:04.549644+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5102,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40468812,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4877","reported_seq":"6981","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.412461+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:08.412461+0000","last_peered":"2021-01-15T18:44:08.412461+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:08.412461+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4877,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38191270,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":624036140,"num_objects":76191,"num_object_clones":0,"num_object_copies":304764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":78534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6884,"num_bytes_recovered":55861248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189806080,"data_stored":1189806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78561,"ondisk_log_size":78561,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215159,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463780,"kb_used_data":305532,"kb_used_omap":0,"kb_used_meta":158207,"kb_avail":93908060,"statfs":{"total":96636764160,"available":96161853440,"internally_reserved":0,"allocated":312864768,"data_stored":312660155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162004020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,12,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":9.5690000000000008},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":10.07}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":10.183999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":9.5120000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.831},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":9.6850000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454244,"kb_used_data":302452,"kb_used_omap":0,"kb_used_meta":151744,"kb_avail":93917596,"statfs":{"total":96636764160,"available":96171618304,"internally_reserved":0,"allocated":309710848,"data_stored":309506235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":155385856},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,5,8,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.8600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":10.396000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":4.6589999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.9730000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456152,"kb_used_data":306468,"kb_used_omap":0,"kb_used_meta":149631,"kb_avail":93915688,"statfs":{"total":96636764160,"available":96169664512,"internally_reserved":0,"allocated":313823232,"data_stored":313618619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153222196},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":2.2010000000000001},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":14.942}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.0110000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":10.68},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":3.1760000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393244,"kb_used_data":249680,"kb_used_omap":0,"kb_used_meta":143551,"kb_avail":93978596,"statfs":{"total":96636764160,"available":96234082304,"internally_reserved":0,"allocated":255672320,"data_stored":255467707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":146996276},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.069}]}]}]}} 2021-01-15T18:44:13.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:14.465 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:14.490 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:14.507 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2021-01-15T18:44:12.484602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616180012,"num_objects":75232,"num_object_clones":0,"num_object_copies":300928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79493,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6911,"num_bytes_recovered":55861248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79520,"ondisk_log_size":79520,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1759656,"kb_used_data":1153672,"kb_used_omap":2,"kb_used_meta":605821,"kb_avail":375727704,"statfs":{"total":386547056640,"available":384745168896,"internally_reserved":0,"allocated":1181360128,"data_stored":1180541676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":620360860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,3,7,14,16,0,0,1,5,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16039912,"num_objects":-1957,"num_object_clones":0,"num_object_copies":-7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1957,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2277,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010770"},"pg_stats":[{"pgid":"1.f","version":"38'4889","reported_seq":"4951","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.660163+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:09.660163+0000","last_peered":"2021-01-15T18:44:09.660163+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:09.660163+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4889,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37838848,"num_objects":4619,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4884","reported_seq":"7805","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:09.754991+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:09.754991+0000","last_peered":"2021-01-15T18:44:09.754991+0000","last_clean":"2021-01-15T18:44:09.754991+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:09.754991+0000","last_undegraded":"2021-01-15T18:44:09.754991+0000","last_fullsized":"2021-01-15T18:44:09.754991+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4884,"ondisk_log_size":4884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37494950,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4873","reported_seq":"4965","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.303523+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:08.303523+0000","last_peered":"2021-01-15T18:44:08.303523+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:08.303523+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4873,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37834859,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4872,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'5021","reported_seq":"5072","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.626021+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:09.626021+0000","last_peered":"2021-01-15T18:44:09.626021+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:09.626021+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5021,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39084032,"num_objects":4771,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'5076","reported_seq":"5666","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:09.713851+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:09.713851+0000","last_peered":"2021-01-15T18:44:09.713851+0000","last_clean":"2021-01-15T18:44:09.713851+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:09.713851+0000","last_undegraded":"2021-01-15T18:44:09.713851+0000","last_fullsized":"2021-01-15T18:44:09.713851+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5076,"ondisk_log_size":5076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38928550,"num_objects":4753,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5074,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4879","reported_seq":"5088","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:09.806965+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:09.806965+0000","last_peered":"2021-01-15T18:44:09.806965+0000","last_clean":"2021-01-15T18:44:09.806965+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:09.806965+0000","last_undegraded":"2021-01-15T18:44:09.806965+0000","last_fullsized":"2021-01-15T18:44:09.806965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4879,"ondisk_log_size":4879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38068390,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4877,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'4996","reported_seq":"5064","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.583012+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:09.583012+0000","last_peered":"2021-01-15T18:44:09.583012+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:09.583012+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4996,"ondisk_log_size":4996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38740134,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5043","reported_seq":"5127","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.175014+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:08.175014+0000","last_peered":"2021-01-15T18:44:08.175014+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:08.175014+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5043,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39432275,"num_objects":4814,"num_object_clones":0,"num_object_copies":19256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'4996","reported_seq":"5052","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.649446+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:09.649446+0000","last_peered":"2021-01-15T18:44:09.649446+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:09.649446+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4996,"ondisk_log_size":4996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38768723,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4995,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'5017","reported_seq":"7956","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:09.552407+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:09.552407+0000","last_peered":"2021-01-15T18:44:09.552407+0000","last_clean":"2021-01-15T18:44:09.552407+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:09.552407+0000","last_undegraded":"2021-01-15T18:44:09.552407+0000","last_fullsized":"2021-01-15T18:44:09.552407+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5017,"ondisk_log_size":5017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38834425,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4916","reported_seq":"5000","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.656302+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:09.656302+0000","last_peered":"2021-01-15T18:44:09.656302+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:09.656302+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4916,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37994662,"num_objects":4639,"num_object_clones":0,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4920","reported_seq":"4992","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.053451+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:09.053451+0000","last_peered":"2021-01-15T18:44:09.053451+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:09.053451+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4920,"ondisk_log_size":4920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38068390,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4877","reported_seq":"7770","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:09.110593+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:09.110593+0000","last_peered":"2021-01-15T18:44:09.110593+0000","last_clean":"2021-01-15T18:44:09.110593+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:09.110593+0000","last_undegraded":"2021-01-15T18:44:09.110593+0000","last_fullsized":"2021-01-15T18:44:09.110593+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4877,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37666982,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5079","reported_seq":"5148","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.182351+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:09.182351+0000","last_peered":"2021-01-15T18:44:09.182351+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:09.182351+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5079,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39379110,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5177","reported_seq":"5255","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:09.360295+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:09.360295+0000","last_peered":"2021-01-15T18:44:09.360295+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:09.360295+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5177,"ondisk_log_size":5177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39854412,"num_objects":4867,"num_object_clones":0,"num_object_copies":19468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4877","reported_seq":"6981","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:08.412461+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:08.412461+0000","last_peered":"2021-01-15T18:44:08.412461+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:08.412461+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4877,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38191270,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4875,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":616180012,"num_objects":75232,"num_object_clones":0,"num_object_copies":300928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":79493,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6911,"num_bytes_recovered":55861248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178927104,"data_stored":1178927104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79520,"ondisk_log_size":79520,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215160,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":453372,"kb_used_data":301008,"kb_used_omap":0,"kb_used_meta":152319,"kb_avail":93918468,"statfs":{"total":96636764160,"available":96172511232,"internally_reserved":0,"allocated":308232192,"data_stored":308027579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":155974708},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,2,4,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.238},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.569}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":1.4690000000000001},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":0.88600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":2.0779999999999998},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.3360000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454244,"kb_used_data":302452,"kb_used_omap":0,"kb_used_meta":151744,"kb_avail":93917596,"statfs":{"total":96636764160,"available":96171618304,"internally_reserved":0,"allocated":309710848,"data_stored":309506235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":155385856},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,5,8,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":1.8600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":10.396000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":4.6589999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.9730000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280571,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":301812,"kb_used_omap":0,"kb_used_meta":164223,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":309055488,"data_stored":308850875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":168164404},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,6,4,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":7.4279999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.294}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":11.616},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":2.2719999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.3120000000000003},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":2.3650000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385956,"kb_used_data":248400,"kb_used_omap":0,"kb_used_meta":137535,"kb_avail":93985884,"statfs":{"total":96636764160,"available":96241545216,"internally_reserved":0,"allocated":254361600,"data_stored":254156987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":140835892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.5920000000000005},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":3.577}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":7.0949999999999998},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":3.7599999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.847},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":3.6749999999999998}]}]}]}} 2021-01-15T18:44:14.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:15.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:15.795 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14366/299208 objects degraded (4.801%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":306,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74802,"data_bytes":612657452,"bytes_used":1797185536,"bytes_avail":384749871104,"bytes_total":386547056640,"degraded_objects":14366,"degraded_total":299208,"degraded_ratio":0.048013422101013344,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":231222,"recovering_keys_per_sec":0,"num_objects_recovered":408,"num_bytes_recovered":2777088,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":108534,"read_op_per_sec":0,"write_op_per_sec":225},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:15.837 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:15.837 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:16.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:16.780 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2021-01-15T18:44:16.485522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607045932,"num_objects":74117,"num_object_clones":0,"num_object_copies":296468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80608,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":56819712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80635,"ondisk_log_size":80635,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1774352,"kb_used_data":1138836,"kb_used_omap":2,"kb_used_meta":635389,"kb_avail":375713008,"statfs":{"total":386547056640,"available":384730120192,"internally_reserved":0,"allocated":1166168064,"data_stored":1165349612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":650638492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,5,5,15,4,0,3,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":90,"apply_latency_ms":90,"commit_latency_ns":90000000,"apply_latency_ns":90000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23519232,"num_objects":-2871,"num_object_clones":0,"num_object_copies":-11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2969,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":1851392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010538"},"pg_stats":[{"pgid":"1.f","version":"38'4955","reported_seq":"5021","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.750736+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:14.750736+0000","last_peered":"2021-01-15T18:44:14.750736+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:14.750736+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4955,"ondisk_log_size":4955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37298176,"num_objects":4553,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'4941","reported_seq":"7863","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:14.804756+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:14.804756+0000","last_peered":"2021-01-15T18:44:14.804756+0000","last_clean":"2021-01-15T18:44:14.804756+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:14.804756+0000","last_undegraded":"2021-01-15T18:44:14.804756+0000","last_fullsized":"2021-01-15T18:44:14.804756+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4941,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37028006,"num_objects":4521,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'4963","reported_seq":"5067","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:13.425023+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:13.425023+0000","last_peered":"2021-01-15T18:44:13.425023+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:13.425023+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4963,"ondisk_log_size":4963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37097579,"num_objects":4530,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4962,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'5093","reported_seq":"5147","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.704302+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:14.704302+0000","last_peered":"2021-01-15T18:44:14.704302+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:14.704302+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5093,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38494208,"num_objects":4699,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'5157","reported_seq":"5747","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:14.878164+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:14.878164+0000","last_peered":"2021-01-15T18:44:14.878164+0000","last_clean":"2021-01-15T18:44:14.878164+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:14.878164+0000","last_undegraded":"2021-01-15T18:44:14.878164+0000","last_fullsized":"2021-01-15T18:44:14.878164+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5157,"ondisk_log_size":5157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38264998,"num_objects":4672,"num_object_clones":0,"num_object_copies":18688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'4953","reported_seq":"5162","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:14.878874+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:14.878874+0000","last_peered":"2021-01-15T18:44:14.878874+0000","last_clean":"2021-01-15T18:44:14.878874+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:14.878874+0000","last_undegraded":"2021-01-15T18:44:14.878874+0000","last_fullsized":"2021-01-15T18:44:14.878874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4953,"ondisk_log_size":4953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37462182,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'5060","reported_seq":"5132","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.717464+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:14.717464+0000","last_peered":"2021-01-15T18:44:14.717464+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:14.717464+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5060,"ondisk_log_size":5060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38215846,"num_objects":4666,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5058,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5106","reported_seq":"5200","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:13.351850+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:13.351850+0000","last_peered":"2021-01-15T18:44:13.351850+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:13.351850+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5106,"ondisk_log_size":5106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38916179,"num_objects":4751,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'5067","reported_seq":"5139","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.706793+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:14.706793+0000","last_peered":"2021-01-15T18:44:14.706793+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:14.706793+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5067,"ondisk_log_size":5067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38187091,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3240}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'5076","reported_seq":"8013","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:14.763962+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:14.763962+0000","last_peered":"2021-01-15T18:44:14.763962+0000","last_clean":"2021-01-15T18:44:14.763962+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:14.763962+0000","last_undegraded":"2021-01-15T18:44:14.763962+0000","last_fullsized":"2021-01-15T18:44:14.763962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5076,"ondisk_log_size":5076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38351097,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'4980","reported_seq":"5066","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.676238+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:14.676238+0000","last_peered":"2021-01-15T18:44:14.676238+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:14.676238+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4980,"ondisk_log_size":4980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37470374,"num_objects":4575,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4978,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'4977","reported_seq":"5049","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.036428+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:14.036428+0000","last_peered":"2021-01-15T18:44:14.036428+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:14.036428+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4977,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37601446,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'4946","reported_seq":"7839","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:14.253959+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:14.253959+0000","last_peered":"2021-01-15T18:44:14.253959+0000","last_clean":"2021-01-15T18:44:14.253959+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:14.253959+0000","last_undegraded":"2021-01-15T18:44:14.253959+0000","last_fullsized":"2021-01-15T18:44:14.253959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4946,"ondisk_log_size":4946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37101734,"num_objects":4530,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5156","reported_seq":"5227","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.183546+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:14.183546+0000","last_peered":"2021-01-15T18:44:14.183546+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:14.183546+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5156,"ondisk_log_size":5156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38748326,"num_objects":4731,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5154,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5254","reported_seq":"5340","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:14.343079+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:14.343079+0000","last_peered":"2021-01-15T18:44:14.343079+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:14.343079+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5254,"ondisk_log_size":5254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39223628,"num_objects":4790,"num_object_clones":0,"num_object_copies":19160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'4951","reported_seq":"7289","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:13.468403+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:13.468403+0000","last_peered":"2021-01-15T18:44:13.468403+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:13.468403+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":4951,"ondisk_log_size":4951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37585062,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4949,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4332}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":607045932,"num_objects":74117,"num_object_clones":0,"num_object_copies":296468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":80608,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":56819712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1164021760,"data_stored":1164021760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80635,"ondisk_log_size":80635,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215161,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":459324,"kb_used_data":296524,"kb_used_omap":0,"kb_used_meta":162751,"kb_avail":93912516,"statfs":{"total":96636764160,"available":96166416384,"internally_reserved":0,"allocated":303640576,"data_stored":303435963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166657076},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,0,0,3,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":1.278},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":4.2850000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.7509999999999999},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460112,"kb_used_data":297820,"kb_used_omap":0,"kb_used_meta":162240,"kb_avail":93911728,"statfs":{"total":96636764160,"available":96165609472,"internally_reserved":0,"allocated":304967680,"data_stored":304763067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":166133760},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,2,5,4,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.93899999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":1.0940000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":1.1830000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.2789999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.532}]}]},{"osd":0,"up_from":22,"seq":94489280572,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455624,"kb_used_data":297208,"kb_used_omap":0,"kb_used_meta":158399,"kb_avail":93916216,"statfs":{"total":96636764160,"available":96170205184,"internally_reserved":0,"allocated":304340992,"data_stored":304136379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162200628},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":0.58499999999999996},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.44900000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":0.61699999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399292,"kb_used_data":247284,"kb_used_omap":0,"kb_used_meta":151999,"kb_avail":93972548,"statfs":{"total":96636764160,"available":96227889152,"internally_reserved":0,"allocated":253218816,"data_stored":253014203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":155647028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":4.024},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":5.7889999999999997},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":5.8869999999999996}]}]}]}} 2021-01-15T18:44:19.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:20.924 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:20.952 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:20.986 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2021-01-15T18:44:20.493289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598223148,"num_objects":73040,"num_object_clones":0,"num_object_copies":292160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":81685,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7249,"num_bytes_recovered":57991168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81712,"ondisk_log_size":81712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1736360,"kb_used_data":1124952,"kb_used_omap":2,"kb_used_meta":611261,"kb_avail":375751000,"statfs":{"total":386547056640,"available":384769024000,"internally_reserved":0,"allocated":1151950848,"data_stored":1151132396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":625931420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,6,5,3,4,0,0,0,4,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25812992,"num_objects":-3151,"num_object_clones":0,"num_object_copies":-12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012620"},"pg_stats":[{"pgid":"1.f","version":"38'5038","reported_seq":"5124","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675795+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:19.675795+0000","last_peered":"2021-01-15T18:44:19.675795+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:19.675795+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5038,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5038,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5022","reported_seq":"7943","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.909047+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:19.909047+0000","last_peered":"2021-01-15T18:44:19.909047+0000","last_clean":"2021-01-15T18:44:19.909047+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:19.909047+0000","last_undegraded":"2021-01-15T18:44:19.909047+0000","last_fullsized":"2021-01-15T18:44:19.909047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5022,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36364454,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5020,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5032","reported_seq":"5149","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.473056+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:18.473056+0000","last_peered":"2021-01-15T18:44:18.473056+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:18.473056+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5032,"ondisk_log_size":5032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36532331,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5031,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'5173","reported_seq":"5237","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675976+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:19.675976+0000","last_peered":"2021-01-15T18:44:19.675976+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:19.675976+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5173,"ondisk_log_size":5173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37838848,"num_objects":4619,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'5222","reported_seq":"5812","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.878170+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:19.878170+0000","last_peered":"2021-01-15T18:44:19.878170+0000","last_clean":"2021-01-15T18:44:19.878170+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:19.878170+0000","last_undegraded":"2021-01-15T18:44:19.878170+0000","last_fullsized":"2021-01-15T18:44:19.878170+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5222,"ondisk_log_size":5222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37732518,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'5002","reported_seq":"5211","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.826645+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:19.826645+0000","last_peered":"2021-01-15T18:44:19.826645+0000","last_clean":"2021-01-15T18:44:19.826645+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:19.826645+0000","last_undegraded":"2021-01-15T18:44:19.826645+0000","last_fullsized":"2021-01-15T18:44:19.826645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5002,"ondisk_log_size":5002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5000,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'5123","reported_seq":"5201","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675873+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:19.675873+0000","last_peered":"2021-01-15T18:44:19.675873+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:19.675873+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5123,"ondisk_log_size":5123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37699750,"num_objects":4603,"num_object_clones":0,"num_object_copies":18412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5121,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5172","reported_seq":"5270","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.421107+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:18.421107+0000","last_peered":"2021-01-15T18:44:18.421107+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:18.421107+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5172,"ondisk_log_size":5172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38375507,"num_objects":4685,"num_object_clones":0,"num_object_copies":18740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'5131","reported_seq":"5216","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.588136+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:19.588136+0000","last_peered":"2021-01-15T18:44:19.588136+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:19.588136+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5131,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37662803,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'5147","reported_seq":"8086","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.686771+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:19.686771+0000","last_peered":"2021-01-15T18:44:19.686771+0000","last_clean":"2021-01-15T18:44:19.686771+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:19.686771+0000","last_undegraded":"2021-01-15T18:44:19.686771+0000","last_fullsized":"2021-01-15T18:44:19.686771+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5147,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37769465,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'5044","reported_seq":"5138","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.623601+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:19.623601+0000","last_peered":"2021-01-15T18:44:19.623601+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:19.623601+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5044,"ondisk_log_size":5044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36946086,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'5043","reported_seq":"5117","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.292487+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:19.292487+0000","last_peered":"2021-01-15T18:44:19.292487+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:19.292487+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5043,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'5003","reported_seq":"7896","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.129977+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:19.129977+0000","last_peered":"2021-01-15T18:44:19.129977+0000","last_clean":"2021-01-15T18:44:19.129977+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:19.129977+0000","last_undegraded":"2021-01-15T18:44:19.129977+0000","last_fullsized":"2021-01-15T18:44:19.129977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5003,"ondisk_log_size":5003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36634790,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5225","reported_seq":"5302","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.141044+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:19.141044+0000","last_peered":"2021-01-15T18:44:19.141044+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:19.141044+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5225,"ondisk_log_size":5225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38183078,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5320","reported_seq":"5418","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.704867+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:19.704867+0000","last_peered":"2021-01-15T18:44:19.704867+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:19.704867+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5320,"ondisk_log_size":5320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38682956,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5015","reported_seq":"7639","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.449455+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:18.449455+0000","last_peered":"2021-01-15T18:44:18.449455+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:18.449455+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5015,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":10387456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":598223148,"num_objects":73040,"num_object_clones":0,"num_object_copies":292160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":81685,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7249,"num_bytes_recovered":57991168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1149812736,"data_stored":1149812736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81712,"ondisk_log_size":81712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448944,"kb_used_data":292144,"kb_used_omap":0,"kb_used_meta":156735,"kb_avail":93922896,"statfs":{"total":96636764160,"available":96177045504,"internally_reserved":0,"allocated":299155456,"data_stored":298950843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160496692},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.157},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":0.91000000000000003},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":2.8460000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.5489999999999999},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":7.6369999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449824,"kb_used_data":293576,"kb_used_omap":0,"kb_used_meta":156224,"kb_avail":93922016,"statfs":{"total":96636764160,"available":96176144384,"internally_reserved":0,"allocated":300621824,"data_stored":300417211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":159973376},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.621}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.55500000000000005}]}]},{"osd":0,"up_from":22,"seq":94489280573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445320,"kb_used_data":292908,"kb_used_omap":0,"kb_used_meta":152383,"kb_avail":93926520,"statfs":{"total":96636764160,"available":96180756480,"internally_reserved":0,"allocated":299937792,"data_stored":299733179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":156040244},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":1.7749999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.6839999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":13.337999999999999},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.399}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.294}]}]},{"osd":1,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392272,"kb_used_data":246324,"kb_used_omap":0,"kb_used_meta":145919,"kb_avail":93979568,"statfs":{"total":96636764160,"available":96235077632,"internally_reserved":0,"allocated":252235776,"data_stored":252031163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":149421108},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.079},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":3.8119999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":5.4130000000000003},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.5529999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.97599999999999998}]}]}]}} 2021-01-15T18:44:20.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:22.113 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:22.114 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14158/292160 objects degraded (4.846%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":312,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73040,"data_bytes":598223148,"bytes_used":1778032640,"bytes_avail":384769024000,"bytes_total":386547056640,"degraded_objects":14158,"degraded_total":292160,"degraded_ratio":0.048459748083242056,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":177306,"recovering_keys_per_sec":0,"num_objects_recovered":365,"num_bytes_recovered":2129920,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:22.205 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:22.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:22.877 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:22.881 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:22.896 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2021-01-15T18:44:22.493808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598223148,"num_objects":73040,"num_object_clones":0,"num_object_copies":292160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":81685,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7249,"num_bytes_recovered":57991168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81712,"ondisk_log_size":81712,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1736360,"kb_used_data":1124952,"kb_used_omap":2,"kb_used_meta":611261,"kb_avail":375751000,"statfs":{"total":386547056640,"available":384769024000,"internally_reserved":0,"allocated":1151950848,"data_stored":1151132396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":625931420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,6,5,3,4,0,0,0,4,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17956864,"num_objects":-2192,"num_object_clones":0,"num_object_copies":-8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2192,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012604"},"pg_stats":[{"pgid":"1.f","version":"38'5038","reported_seq":"5124","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675795+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:19.675795+0000","last_peered":"2021-01-15T18:44:19.675795+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:19.675795+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5038,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5038,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"38'5022","reported_seq":"7943","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.909047+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:19.909047+0000","last_peered":"2021-01-15T18:44:19.909047+0000","last_clean":"2021-01-15T18:44:19.909047+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:19.909047+0000","last_undegraded":"2021-01-15T18:44:19.909047+0000","last_fullsized":"2021-01-15T18:44:19.909047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5022,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36364454,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5020,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5032","reported_seq":"5149","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.473056+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:18.473056+0000","last_peered":"2021-01-15T18:44:18.473056+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:18.473056+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5032,"ondisk_log_size":5032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36532331,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5031,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"38'5173","reported_seq":"5237","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675976+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:19.675976+0000","last_peered":"2021-01-15T18:44:19.675976+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:19.675976+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5173,"ondisk_log_size":5173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37838848,"num_objects":4619,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"38'5222","reported_seq":"5812","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.878170+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:19.878170+0000","last_peered":"2021-01-15T18:44:19.878170+0000","last_clean":"2021-01-15T18:44:19.878170+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:19.878170+0000","last_undegraded":"2021-01-15T18:44:19.878170+0000","last_fullsized":"2021-01-15T18:44:19.878170+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5222,"ondisk_log_size":5222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37732518,"num_objects":4607,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"38'5002","reported_seq":"5211","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.826645+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:19.826645+0000","last_peered":"2021-01-15T18:44:19.826645+0000","last_clean":"2021-01-15T18:44:19.826645+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:19.826645+0000","last_undegraded":"2021-01-15T18:44:19.826645+0000","last_fullsized":"2021-01-15T18:44:19.826645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5002,"ondisk_log_size":5002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5000,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"38'5123","reported_seq":"5201","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.675873+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:19.675873+0000","last_peered":"2021-01-15T18:44:19.675873+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:19.675873+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5123,"ondisk_log_size":5123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37699750,"num_objects":4603,"num_object_clones":0,"num_object_copies":18412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5121,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5172","reported_seq":"5270","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.421107+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:18.421107+0000","last_peered":"2021-01-15T18:44:18.421107+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:18.421107+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5172,"ondisk_log_size":5172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38375507,"num_objects":4685,"num_object_clones":0,"num_object_copies":18740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"38'5131","reported_seq":"5216","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.588136+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:19.588136+0000","last_peered":"2021-01-15T18:44:19.588136+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:19.588136+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5131,"ondisk_log_size":5131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37662803,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"38'5147","reported_seq":"8086","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.686771+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:19.686771+0000","last_peered":"2021-01-15T18:44:19.686771+0000","last_clean":"2021-01-15T18:44:19.686771+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:19.686771+0000","last_undegraded":"2021-01-15T18:44:19.686771+0000","last_fullsized":"2021-01-15T18:44:19.686771+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5147,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37769465,"num_objects":4612,"num_object_clones":0,"num_object_copies":18448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5144,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"38'5044","reported_seq":"5138","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.623601+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:19.623601+0000","last_peered":"2021-01-15T18:44:19.623601+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:19.623601+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5044,"ondisk_log_size":5044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36946086,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"38'5043","reported_seq":"5117","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.292487+0000","last_change":"2021-01-15T18:40:32.783689+0000","last_active":"2021-01-15T18:44:19.292487+0000","last_peered":"2021-01-15T18:44:19.292487+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:19.292487+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5043,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3158}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"38'5003","reported_seq":"7896","reported_epoch":"38","state":"active+clean","last_fresh":"2021-01-15T18:44:19.129977+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:19.129977+0000","last_peered":"2021-01-15T18:44:19.129977+0000","last_clean":"2021-01-15T18:44:19.129977+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:19.129977+0000","last_undegraded":"2021-01-15T18:44:19.129977+0000","last_fullsized":"2021-01-15T18:44:19.129977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5003,"ondisk_log_size":5003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36634790,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"38'5225","reported_seq":"5302","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.141044+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:19.141044+0000","last_peered":"2021-01-15T18:44:19.141044+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:19.141044+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5225,"ondisk_log_size":5225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38183078,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"38'5320","reported_seq":"5418","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:19.704867+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:19.704867+0000","last_peered":"2021-01-15T18:44:19.704867+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:19.704867+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5320,"ondisk_log_size":5320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38682956,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5015","reported_seq":"7639","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:18.449455+0000","last_change":"2021-01-15T18:43:07.740356+0000","last_active":"2021-01-15T18:44:18.449455+0000","last_peered":"2021-01-15T18:44:18.449455+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:18.449455+0000","last_undegraded":"2021-01-15T18:40:32.514272+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5015,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060774,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":10387456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":598223148,"num_objects":73040,"num_object_clones":0,"num_object_copies":292160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":81685,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7249,"num_bytes_recovered":57991168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1149812736,"data_stored":1149812736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81712,"ondisk_log_size":81712,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448944,"kb_used_data":292144,"kb_used_omap":0,"kb_used_meta":156735,"kb_avail":93922896,"statfs":{"total":96636764160,"available":96177045504,"internally_reserved":0,"allocated":299155456,"data_stored":298950843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160496692},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,4,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.157},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":0.91000000000000003},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":2.8460000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.5489999999999999},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":7.6369999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449824,"kb_used_data":293576,"kb_used_omap":0,"kb_used_meta":156224,"kb_avail":93922016,"statfs":{"total":96636764160,"available":96176144384,"internally_reserved":0,"allocated":300621824,"data_stored":300417211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":159973376},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.621}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.55500000000000005}]}]},{"osd":0,"up_from":22,"seq":94489280573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445320,"kb_used_data":292908,"kb_used_omap":0,"kb_used_meta":152383,"kb_avail":93926520,"statfs":{"total":96636764160,"available":96180756480,"internally_reserved":0,"allocated":299937792,"data_stored":299733179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":156040244},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":1.7749999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.6839999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":13.337999999999999},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.399}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.294}]}]},{"osd":1,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392272,"kb_used_data":246324,"kb_used_omap":0,"kb_used_meta":145919,"kb_avail":93979568,"statfs":{"total":96636764160,"available":96235077632,"internally_reserved":0,"allocated":252235776,"data_stored":252031163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":149421108},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.079},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":3.8119999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":5.4130000000000003},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":3.5529999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.97599999999999998}]}]}]}} 2021-01-15T18:44:25.898 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:26.993 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:27.012 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:27.047 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2021-01-15T18:44:26.495346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589621548,"num_objects":71990,"num_object_clones":0,"num_object_copies":287960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":82735,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":59162624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82762,"ondisk_log_size":82762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1759940,"kb_used_data":1111576,"kb_used_omap":2,"kb_used_meta":652349,"kb_avail":375727420,"statfs":{"total":386547056640,"available":384744878080,"internally_reserved":0,"allocated":1138253824,"data_stored":1137416263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":668005532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,8,10,5,0,0,0,5,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17424384,"num_objects":-2127,"num_object_clones":0,"num_object_copies":-8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":2342912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002525"},"pg_stats":[{"pgid":"1.f","version":"40'5090","reported_seq":"5190","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.671104+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:24.671104+0000","last_peered":"2021-01-15T18:44:24.671104+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:24.671104+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5090,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5090,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3089}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5085","reported_seq":"8009","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.848682+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:24.848682+0000","last_peered":"2021-01-15T18:44:24.848682+0000","last_clean":"2021-01-15T18:44:24.848682+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:24.848682+0000","last_undegraded":"2021-01-15T18:44:24.848682+0000","last_fullsized":"2021-01-15T18:44:24.848682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5085,"ondisk_log_size":5085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35848358,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5101","reported_seq":"5219","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:23.342419+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:23.342419+0000","last_peered":"2021-01-15T18:44:23.342419+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:23.342419+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5101,"ondisk_log_size":5101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35967083,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5100,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2994}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5242","reported_seq":"5320","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.659037+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:24.659037+0000","last_peered":"2021-01-15T18:44:24.659037+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:24.659037+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5242,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37273600,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5285","reported_seq":"5877","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.668967+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:24.668967+0000","last_peered":"2021-01-15T18:44:24.668967+0000","last_clean":"2021-01-15T18:44:24.668967+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:24.668967+0000","last_undegraded":"2021-01-15T18:44:24.668967+0000","last_fullsized":"2021-01-15T18:44:24.668967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5285,"ondisk_log_size":5285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37216422,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5087","reported_seq":"5298","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.848016+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:24.848016+0000","last_peered":"2021-01-15T18:44:24.848016+0000","last_clean":"2021-01-15T18:44:24.848016+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:24.848016+0000","last_undegraded":"2021-01-15T18:44:24.848016+0000","last_fullsized":"2021-01-15T18:44:24.848016+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5087,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36364454,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'5175","reported_seq":"5265","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.608904+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:24.608904+0000","last_peered":"2021-01-15T18:44:24.608904+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:24.608904+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5175,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37273766,"num_objects":4551,"num_object_clones":0,"num_object_copies":18204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5247","reported_seq":"5344","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:23.477614+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:23.477614+0000","last_peered":"2021-01-15T18:44:23.477614+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:23.477614+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5247,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37761107,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5246,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5194","reported_seq":"5286","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.779644+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:24.779644+0000","last_peered":"2021-01-15T18:44:24.779644+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:24.779644+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5194,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37146707,"num_objects":4535,"num_object_clones":0,"num_object_copies":18140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5209","reported_seq":"8150","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.652408+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:24.652408+0000","last_peered":"2021-01-15T18:44:24.652408+0000","last_clean":"2021-01-15T18:44:24.652408+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:24.652408+0000","last_undegraded":"2021-01-15T18:44:24.652408+0000","last_fullsized":"2021-01-15T18:44:24.652408+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5209,"ondisk_log_size":5209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37261561,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5106","reported_seq":"5218","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.654281+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:24.654281+0000","last_peered":"2021-01-15T18:44:24.654281+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:24.654281+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5106,"ondisk_log_size":5106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36438182,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"39'5107","reported_seq":"5243","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.313807+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:24.313807+0000","last_peered":"2021-01-15T18:44:24.313807+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:24.313807+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5107,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36536486,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"39'5075","reported_seq":"7969","reported_epoch":"39","state":"active+clean","last_fresh":"2021-01-15T18:44:24.081796+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:24.081796+0000","last_peered":"2021-01-15T18:44:24.081796+0000","last_clean":"2021-01-15T18:44:24.081796+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:24.081796+0000","last_undegraded":"2021-01-15T18:44:24.081796+0000","last_fullsized":"2021-01-15T18:44:24.081796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5075,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36044966,"num_objects":4401,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'5298","reported_seq":"5382","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.257179+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:24.257179+0000","last_peered":"2021-01-15T18:44:24.257179+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:24.257179+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5298,"ondisk_log_size":5298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37585062,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5296,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'5386","reported_seq":"5500","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.636192+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:24.636192+0000","last_peered":"2021-01-15T18:44:24.636192+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:24.636192+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5386,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38142284,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5075","reported_seq":"7928","reported_epoch":"38","state":"active+undersized+remapped","last_fresh":"2021-01-15T18:44:23.152896+0000","last_change":"2021-01-15T18:44:22.912871+0000","last_active":"2021-01-15T18:44:23.152896+0000","last_peered":"2021-01-15T18:44:23.152896+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:23.152896+0000","last_undegraded":"2021-01-15T18:44:23.152896+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5075,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36569254,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","1(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4465}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":589621548,"num_objects":71990,"num_object_clones":0,"num_object_copies":287960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":82735,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":59162624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1135505408,"data_stored":1135505408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82762,"ondisk_log_size":82762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455376,"kb_used_data":288568,"kb_used_omap":0,"kb_used_meta":166783,"kb_avail":93916464,"statfs":{"total":96636764160,"available":96170459136,"internally_reserved":0,"allocated":295493632,"data_stored":295284247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170785844},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.788}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":30.129000000000001},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":24.998999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.505},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.8140000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460000,"kb_used_data":289256,"kb_used_omap":0,"kb_used_meta":174784,"kb_avail":93911840,"statfs":{"total":96636764160,"available":96165724160,"internally_reserved":0,"allocated":296198144,"data_stored":295993531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":178978816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":14.413},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.61399999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.3540000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280575,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455524,"kb_used_data":288656,"kb_used_omap":0,"kb_used_meta":166847,"kb_avail":93916316,"statfs":{"total":96636764160,"available":96170307584,"internally_reserved":0,"allocated":295583744,"data_stored":295374359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170851380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":1.7749999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.6839999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":13.337999999999999},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.399}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.294}]}]},{"osd":1,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389040,"kb_used_data":245096,"kb_used_omap":0,"kb_used_meta":143935,"kb_avail":93982800,"statfs":{"total":96636764160,"available":96238387200,"internally_reserved":0,"allocated":250978304,"data_stored":250764126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147389492},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.046},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.82899999999999996}]}]}]}} 2021-01-15T18:44:27.049 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:28.526 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:28.527 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13975/287960 objects degraded (4.853%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71990,"data_bytes":589621548,"bytes_used":1802178560,"bytes_avail":384744878080,"bytes_total":386547056640,"degraded_objects":13975,"degraded_total":287960,"degraded_ratio":0.048531045978608141,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":292771,"recovering_keys_per_sec":0,"num_objects_recovered":371,"num_bytes_recovered":2342912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":265},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:28.574 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:28.575 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:29.310 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:29.315 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:29.360 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2021-01-15T18:44:28.497812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589621548,"num_objects":71990,"num_object_clones":0,"num_object_copies":287960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":82735,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":59162624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82762,"ondisk_log_size":82762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1759940,"kb_used_data":1111576,"kb_used_omap":2,"kb_used_meta":652349,"kb_avail":375727420,"statfs":{"total":386547056640,"available":384744878080,"internally_reserved":0,"allocated":1138253824,"data_stored":1137416263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":668005532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,8,10,5,0,0,0,5,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8601600,"num_objects":-1050,"num_object_clones":0,"num_object_copies":-4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004522"},"pg_stats":[{"pgid":"1.f","version":"40'5090","reported_seq":"5190","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.671104+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:24.671104+0000","last_peered":"2021-01-15T18:44:24.671104+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:24.671104+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5090,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36192256,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5090,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3089}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5085","reported_seq":"8009","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.848682+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:24.848682+0000","last_peered":"2021-01-15T18:44:24.848682+0000","last_clean":"2021-01-15T18:44:24.848682+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:24.848682+0000","last_undegraded":"2021-01-15T18:44:24.848682+0000","last_fullsized":"2021-01-15T18:44:24.848682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5085,"ondisk_log_size":5085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35848358,"num_objects":4377,"num_object_clones":0,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"38'5101","reported_seq":"5219","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:23.342419+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:23.342419+0000","last_peered":"2021-01-15T18:44:23.342419+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:23.342419+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5101,"ondisk_log_size":5101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35967083,"num_objects":4392,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5100,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2994}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5242","reported_seq":"5320","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.659037+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:24.659037+0000","last_peered":"2021-01-15T18:44:24.659037+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:24.659037+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5242,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37273600,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5285","reported_seq":"5877","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.668967+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:24.668967+0000","last_peered":"2021-01-15T18:44:24.668967+0000","last_clean":"2021-01-15T18:44:24.668967+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:24.668967+0000","last_undegraded":"2021-01-15T18:44:24.668967+0000","last_fullsized":"2021-01-15T18:44:24.668967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5285,"ondisk_log_size":5285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37216422,"num_objects":4544,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5087","reported_seq":"5298","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.848016+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:24.848016+0000","last_peered":"2021-01-15T18:44:24.848016+0000","last_clean":"2021-01-15T18:44:24.848016+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:24.848016+0000","last_undegraded":"2021-01-15T18:44:24.848016+0000","last_fullsized":"2021-01-15T18:44:24.848016+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5087,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36364454,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"39'5175","reported_seq":"5265","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.608904+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:24.608904+0000","last_peered":"2021-01-15T18:44:24.608904+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:24.608904+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5175,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37273766,"num_objects":4551,"num_object_clones":0,"num_object_copies":18204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3175}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"38'5247","reported_seq":"5344","reported_epoch":"38","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:23.477614+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:23.477614+0000","last_peered":"2021-01-15T18:44:23.477614+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:23.477614+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5247,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37761107,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5246,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5194","reported_seq":"5286","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.779644+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:24.779644+0000","last_peered":"2021-01-15T18:44:24.779644+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:24.779644+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5194,"ondisk_log_size":5194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37146707,"num_objects":4535,"num_object_clones":0,"num_object_copies":18140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3122}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5209","reported_seq":"8150","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:24.652408+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:24.652408+0000","last_peered":"2021-01-15T18:44:24.652408+0000","last_clean":"2021-01-15T18:44:24.652408+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:24.652408+0000","last_undegraded":"2021-01-15T18:44:24.652408+0000","last_fullsized":"2021-01-15T18:44:24.652408+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5209,"ondisk_log_size":5209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37261561,"num_objects":4550,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5106","reported_seq":"5218","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.654281+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:24.654281+0000","last_peered":"2021-01-15T18:44:24.654281+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:24.654281+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5106,"ondisk_log_size":5106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36438182,"num_objects":4449,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"39'5107","reported_seq":"5243","reported_epoch":"39","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.313807+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:24.313807+0000","last_peered":"2021-01-15T18:44:24.313807+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:24.313807+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5107,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36536486,"num_objects":4461,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"39'5075","reported_seq":"7969","reported_epoch":"39","state":"active+clean","last_fresh":"2021-01-15T18:44:24.081796+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:24.081796+0000","last_peered":"2021-01-15T18:44:24.081796+0000","last_clean":"2021-01-15T18:44:24.081796+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:24.081796+0000","last_undegraded":"2021-01-15T18:44:24.081796+0000","last_fullsized":"2021-01-15T18:44:24.081796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5075,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36044966,"num_objects":4401,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"39'5298","reported_seq":"5382","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.257179+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:24.257179+0000","last_peered":"2021-01-15T18:44:24.257179+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:24.257179+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5298,"ondisk_log_size":5298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37585062,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5296,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"39'5386","reported_seq":"5500","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:24.636192+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:24.636192+0000","last_peered":"2021-01-15T18:44:24.636192+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:24.636192+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5386,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38142284,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5383,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"38'5075","reported_seq":"7928","reported_epoch":"38","state":"active+undersized+remapped","last_fresh":"2021-01-15T18:44:23.152896+0000","last_change":"2021-01-15T18:44:22.912871+0000","last_active":"2021-01-15T18:44:23.152896+0000","last_peered":"2021-01-15T18:44:23.152896+0000","last_clean":"2021-01-15T18:39:28.527786+0000","last_became_active":"2021-01-15T18:40:32.744120+0000","last_became_peered":"2021-01-15T18:40:32.744120+0000","last_unstale":"2021-01-15T18:44:23.152896+0000","last_undegraded":"2021-01-15T18:44:23.152896+0000","last_fullsized":"2021-01-15T18:40:32.487597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5075,"ondisk_log_size":5075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36569254,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","1(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":4465}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":589621548,"num_objects":71990,"num_object_clones":0,"num_object_copies":287960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":82735,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":59162624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1135505408,"data_stored":1135505408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82762,"ondisk_log_size":82762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215163,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455376,"kb_used_data":288568,"kb_used_omap":0,"kb_used_meta":166783,"kb_avail":93916464,"statfs":{"total":96636764160,"available":96170459136,"internally_reserved":0,"allocated":295493632,"data_stored":295284247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170785844},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.788}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":30.129000000000001},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":24.998999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.505},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.8140000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":460000,"kb_used_data":289256,"kb_used_omap":0,"kb_used_meta":174784,"kb_avail":93911840,"statfs":{"total":96636764160,"available":96165724160,"internally_reserved":0,"allocated":296198144,"data_stored":295993531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":178978816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":14.413},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.61399999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.3540000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280575,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455524,"kb_used_data":288656,"kb_used_omap":0,"kb_used_meta":166847,"kb_avail":93916316,"statfs":{"total":96636764160,"available":96170307584,"internally_reserved":0,"allocated":295583744,"data_stored":295374359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170851380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":1.7749999999999999},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.6839999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":13.337999999999999},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.399}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.294}]}]},{"osd":1,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389040,"kb_used_data":245096,"kb_used_omap":0,"kb_used_meta":143935,"kb_avail":93982800,"statfs":{"total":96636764160,"available":96238387200,"internally_reserved":0,"allocated":250978304,"data_stored":250764126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147389492},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.046},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.82899999999999996}]}]}]}} 2021-01-15T18:44:32.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:33.630 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:33.637 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:33.663 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2021-01-15T18:44:32.498800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":580258092,"num_objects":70847,"num_object_clones":0,"num_object_copies":283388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":83878,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7554,"num_bytes_recovered":59899904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83905,"ondisk_log_size":83905,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1737424,"kb_used_data":1096436,"kb_used_omap":2,"kb_used_meta":640829,"kb_avail":375749936,"statfs":{"total":386547056640,"available":384767934464,"internally_reserved":0,"allocated":1122750464,"data_stored":1121893752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":656209052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,4,9,15,14,7,1,2,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16293888,"num_objects":-1989,"num_object_clones":0,"num_object_copies":-7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.894914"},"pg_stats":[{"pgid":"1.f","version":"40'5162","reported_seq":"5262","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.765281+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:29.765281+0000","last_peered":"2021-01-15T18:44:29.765281+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:29.765281+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5162,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35602432,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5149","reported_seq":"8073","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.832239+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:29.832239+0000","last_peered":"2021-01-15T18:44:29.832239+0000","last_clean":"2021-01-15T18:44:29.832239+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:29.832239+0000","last_undegraded":"2021-01-15T18:44:29.832239+0000","last_fullsized":"2021-01-15T18:44:29.832239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5149,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35324070,"num_objects":4313,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5147,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5176","reported_seq":"5296","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:28.322834+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:28.322834+0000","last_peered":"2021-01-15T18:44:28.322834+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:28.322834+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5176,"ondisk_log_size":5176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35352683,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5175,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5295","reported_seq":"5379","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.762912+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:29.762912+0000","last_peered":"2021-01-15T18:44:29.762912+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:29.762912+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5295,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5295,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5353","reported_seq":"5945","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.798948+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:29.798948+0000","last_peered":"2021-01-15T18:44:29.798948+0000","last_clean":"2021-01-15T18:44:29.798948+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:29.798948+0000","last_undegraded":"2021-01-15T18:44:29.798948+0000","last_fullsized":"2021-01-15T18:44:29.798948+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5353,"ondisk_log_size":5353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36659366,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5351,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5145","reported_seq":"5356","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.719553+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:29.719553+0000","last_peered":"2021-01-15T18:44:29.719553+0000","last_clean":"2021-01-15T18:44:29.719553+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:29.719553+0000","last_undegraded":"2021-01-15T18:44:29.719553+0000","last_fullsized":"2021-01-15T18:44:29.719553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5145,"ondisk_log_size":5145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35889318,"num_objects":4382,"num_object_clones":0,"num_object_copies":17528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5143,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5253","reported_seq":"5359","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.780899+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:29.780899+0000","last_peered":"2021-01-15T18:44:29.780899+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:29.780899+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5253,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36634790,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5318","reported_seq":"5419","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:28.502797+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:28.502797+0000","last_peered":"2021-01-15T18:44:28.502797+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:28.502797+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5318,"ondisk_log_size":5318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37179475,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5274","reported_seq":"5382","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.785069+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:29.785069+0000","last_peered":"2021-01-15T18:44:29.785069+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:29.785069+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5274,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36491347,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3050}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5279","reported_seq":"8220","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.726316+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:29.726316+0000","last_peered":"2021-01-15T18:44:29.726316+0000","last_clean":"2021-01-15T18:44:29.726316+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:29.726316+0000","last_undegraded":"2021-01-15T18:44:29.726316+0000","last_fullsized":"2021-01-15T18:44:29.726316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5279,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36688121,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5182","reported_seq":"5304","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.701604+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:29.701604+0000","last_peered":"2021-01-15T18:44:29.701604+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:29.701604+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5182,"ondisk_log_size":5182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35815590,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5180,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5188","reported_seq":"5507","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.315841+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:29.315841+0000","last_peered":"2021-01-15T18:44:29.315841+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:29.315841+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5188,"ondisk_log_size":5188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35872934,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5160","reported_seq":"8055","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.315863+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:29.315863+0000","last_peered":"2021-01-15T18:44:29.315863+0000","last_clean":"2021-01-15T18:44:29.315863+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:29.315863+0000","last_undegraded":"2021-01-15T18:44:29.315863+0000","last_fullsized":"2021-01-15T18:44:29.315863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5160,"ondisk_log_size":5160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35348646,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5370","reported_seq":"5459","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.091292+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:29.091292+0000","last_peered":"2021-01-15T18:44:29.091292+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:29.091292+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5370,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36995238,"num_objects":4517,"num_object_clones":0,"num_object_copies":18068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5452","reported_seq":"5576","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.709243+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:29.709243+0000","last_peered":"2021-01-15T18:44:29.709243+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:29.709243+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5452,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37601612,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5149","reported_seq":"8014","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:28.477521+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:28.477521+0000","last_peered":"2021-01-15T18:44:28.477521+0000","last_clean":"2021-01-15T18:44:28.477521+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:28.477521+0000","last_undegraded":"2021-01-15T18:44:28.477521+0000","last_fullsized":"2021-01-15T18:44:28.477521+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5149,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35963046,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5147,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":580258092,"num_objects":70847,"num_object_clones":0,"num_object_copies":283388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":83878,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7554,"num_bytes_recovered":59899904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1119883264,"data_stored":1119883264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83905,"ondisk_log_size":83905,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444980,"kb_used_data":284092,"kb_used_omap":0,"kb_used_meta":160831,"kb_avail":93926860,"statfs":{"total":96636764160,"available":96181104640,"internally_reserved":0,"allocated":290910208,"data_stored":290696030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164690996},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,3,4,4,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":13.646000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":13.542999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":13.856}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":13.433999999999999},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":17.222999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449676,"kb_used_data":284860,"kb_used_omap":0,"kb_used_meta":164800,"kb_avail":93922164,"statfs":{"total":96636764160,"available":96176295936,"internally_reserved":0,"allocated":291696640,"data_stored":291482462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":168755200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,0,1,1,1,2,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":14.413},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":0.66700000000000004},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.615},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":10.016999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280576,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445060,"kb_used_data":284056,"kb_used_omap":0,"kb_used_meta":160959,"kb_avail":93926780,"statfs":{"total":96636764160,"available":96181022720,"internally_reserved":0,"allocated":290873344,"data_stored":290659166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164822068},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:43:32 2021","interfaces":[{"interface":"back","average":{"1min":2.9910000000000001,"5min":2.6400000000000001,"15min":2.839},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.736000000000001,"5min":13.311,"15min":13.311},"last":3.7959999999999998},{"interface":"front","average":{"1min":7.5880000000000001,"5min":3.544,"15min":2.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":44.072000000000003,"5min":44.072000000000003,"15min":44.072000000000003},"last":3.903}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.202},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.70399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":12.702999999999999},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":12.805999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397708,"kb_used_data":243428,"kb_used_omap":0,"kb_used_meta":154239,"kb_avail":93974132,"statfs":{"total":96636764160,"available":96229511168,"internally_reserved":0,"allocated":249270272,"data_stored":249056094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":157940788},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.7269999999999999},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.3150000000000004},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.753}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.077},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":6.133}]}]}]}} 2021-01-15T18:44:33.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:34.856 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:34.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13740/281704 objects degraded (4.877%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70426,"data_bytes":576809260,"bytes_used":1774194688,"bytes_avail":384772861952,"bytes_total":386547056640,"degraded_objects":13740,"degraded_total":281704,"degraded_ratio":0.048774600289665751,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":162950,"recovering_keys_per_sec":0,"num_objects_recovered":269,"num_bytes_recovered":1630208,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:34.904 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:34.905 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:35.687 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:35.743 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2021-01-15T18:44:34.499154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576809260,"num_objects":70426,"num_object_clones":0,"num_object_copies":281704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":84299,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7665,"num_bytes_recovered":60792832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84326,"ondisk_log_size":84326,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1732612,"kb_used_data":1087128,"kb_used_omap":2,"kb_used_meta":645245,"kb_avail":375754748,"statfs":{"total":386547056640,"available":384772861952,"internally_reserved":0,"allocated":1113219072,"data_stored":1112362360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":660731036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,4,9,20,12,9,0,0,3,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18030592,"num_objects":-2201,"num_object_clones":0,"num_object_copies":-8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004287"},"pg_stats":[{"pgid":"1.f","version":"40'5162","reported_seq":"5262","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.765281+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:29.765281+0000","last_peered":"2021-01-15T18:44:29.765281+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:29.765281+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5162,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35602432,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5149","reported_seq":"8073","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.832239+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:29.832239+0000","last_peered":"2021-01-15T18:44:29.832239+0000","last_clean":"2021-01-15T18:44:29.832239+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:29.832239+0000","last_undegraded":"2021-01-15T18:44:29.832239+0000","last_fullsized":"2021-01-15T18:44:29.832239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5149,"ondisk_log_size":5149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35324070,"num_objects":4313,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5147,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5246","reported_seq":"5366","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:33.367346+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:33.367346+0000","last_peered":"2021-01-15T18:44:33.367346+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:33.367346+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5246,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34779243,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5245,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2849}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5295","reported_seq":"5379","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.762912+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:29.762912+0000","last_peered":"2021-01-15T18:44:29.762912+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:29.762912+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5295,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5295,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5353","reported_seq":"5945","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.798948+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:29.798948+0000","last_peered":"2021-01-15T18:44:29.798948+0000","last_clean":"2021-01-15T18:44:29.798948+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:29.798948+0000","last_undegraded":"2021-01-15T18:44:29.798948+0000","last_fullsized":"2021-01-15T18:44:29.798948+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5353,"ondisk_log_size":5353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36659366,"num_objects":4476,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5351,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5145","reported_seq":"5356","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.719553+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:29.719553+0000","last_peered":"2021-01-15T18:44:29.719553+0000","last_clean":"2021-01-15T18:44:29.719553+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:29.719553+0000","last_undegraded":"2021-01-15T18:44:29.719553+0000","last_fullsized":"2021-01-15T18:44:29.719553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5145,"ondisk_log_size":5145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35889318,"num_objects":4382,"num_object_clones":0,"num_object_copies":17528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5143,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5253","reported_seq":"5359","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.780899+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:29.780899+0000","last_peered":"2021-01-15T18:44:29.780899+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:29.780899+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5253,"ondisk_log_size":5253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36634790,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3105}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5392","reported_seq":"5500","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:33.333235+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:33.333235+0000","last_peered":"2021-01-15T18:44:33.333235+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:33.333235+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5392,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36573267,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5274","reported_seq":"5382","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.785069+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:29.785069+0000","last_peered":"2021-01-15T18:44:29.785069+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:29.785069+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5274,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36491347,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3050}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5279","reported_seq":"8220","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:29.726316+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:29.726316+0000","last_peered":"2021-01-15T18:44:29.726316+0000","last_clean":"2021-01-15T18:44:29.726316+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:29.726316+0000","last_undegraded":"2021-01-15T18:44:29.726316+0000","last_fullsized":"2021-01-15T18:44:29.726316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5279,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36688121,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5182","reported_seq":"5304","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.701604+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:29.701604+0000","last_peered":"2021-01-15T18:44:29.701604+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:29.701604+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5182,"ondisk_log_size":5182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35815590,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5180,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5252","reported_seq":"5787","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.296718+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:34.296718+0000","last_peered":"2021-01-15T18:44:34.296718+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:34.296718+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5252,"ondisk_log_size":5252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35348646,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5250,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1867776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5230","reported_seq":"8125","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.240301+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:34.240301+0000","last_peered":"2021-01-15T18:44:34.240301+0000","last_clean":"2021-01-15T18:44:34.240301+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:34.240301+0000","last_undegraded":"2021-01-15T18:44:34.240301+0000","last_fullsized":"2021-01-15T18:44:34.240301+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34775206,"num_objects":4246,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5437","reported_seq":"5526","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.241501+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:34.241501+0000","last_peered":"2021-01-15T18:44:34.241501+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:34.241501+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5437,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36446374,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5452","reported_seq":"5576","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:29.709243+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:29.709243+0000","last_peered":"2021-01-15T18:44:29.709243+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:29.709243+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5452,"ondisk_log_size":5452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37601612,"num_objects":4592,"num_object_clones":0,"num_object_copies":18368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5225","reported_seq":"8089","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:33.518132+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:33.518132+0000","last_peered":"2021-01-15T18:44:33.518132+0000","last_clean":"2021-01-15T18:44:33.518132+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:33.518132+0000","last_undegraded":"2021-01-15T18:44:33.518132+0000","last_fullsized":"2021-01-15T18:44:33.518132+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5225,"ondisk_log_size":5225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35340454,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":576809260,"num_objects":70426,"num_object_clones":0,"num_object_copies":281704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":84299,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7665,"num_bytes_recovered":60792832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110409216,"data_stored":1110409216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84326,"ondisk_log_size":84326,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215164,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444980,"kb_used_data":284092,"kb_used_omap":0,"kb_used_meta":160831,"kb_avail":93926860,"statfs":{"total":96636764160,"available":96181104640,"internally_reserved":0,"allocated":290910208,"data_stored":290696030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164690996},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,3,4,4,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":13.646000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":13.542999999999999},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":13.856}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":13.433999999999999},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":17.222999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439072,"kb_used_data":280216,"kb_used_omap":0,"kb_used_meta":158784,"kb_avail":93932768,"statfs":{"total":96636764160,"available":96187154432,"internally_reserved":0,"allocated":286941184,"data_stored":286727006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":162594816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.752},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":2.452},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":2.5409999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.99299999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.69099999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280577,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450852,"kb_used_data":279392,"kb_used_omap":0,"kb_used_meta":171391,"kb_avail":93920988,"statfs":{"total":96636764160,"available":96175091712,"internally_reserved":0,"allocated":286097408,"data_stored":285883230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":175504436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.0379999999999998},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.7120000000000002},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":5.3140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.4269999999999996},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":5.5279999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397708,"kb_used_data":243428,"kb_used_omap":0,"kb_used_meta":154239,"kb_avail":93974132,"statfs":{"total":96636764160,"available":96229511168,"internally_reserved":0,"allocated":249270272,"data_stored":249056094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":157940788},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.7269999999999999},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.3150000000000004},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.753}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.077},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":6.133}]}]}]}} 2021-01-15T18:44:38.747 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:40.058 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:40.083 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:40.136 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2021-01-15T18:44:38.500241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570738988,"num_objects":69685,"num_object_clones":0,"num_object_copies":278740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85040,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7684,"num_bytes_recovered":60792832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85067,"ondisk_log_size":85067,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1730800,"kb_used_data":1080964,"kb_used_omap":2,"kb_used_meta":649725,"kb_avail":375756560,"statfs":{"total":386547056640,"available":384774717440,"internally_reserved":0,"allocated":1106907136,"data_stored":1106050424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":665318556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,7,20,22,3,0,0,5,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":103,"apply_latency_ms":103,"commit_latency_ns":103000000,"apply_latency_ns":103000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18882560,"num_objects":-2305,"num_object_clones":0,"num_object_copies":-9220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":1630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"40'5220","reported_seq":"5326","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.651279+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:34.651279+0000","last_peered":"2021-01-15T18:44:34.651279+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:34.651279+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5220,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35127296,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2962}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5219","reported_seq":"8142","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.702825+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:34.702825+0000","last_peered":"2021-01-15T18:44:34.702825+0000","last_clean":"2021-01-15T18:44:34.702825+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:34.702825+0000","last_undegraded":"2021-01-15T18:44:34.702825+0000","last_fullsized":"2021-01-15T18:44:34.702825+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5219,"ondisk_log_size":5219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34750630,"num_objects":4243,"num_object_clones":0,"num_object_copies":16972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5217,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5246","reported_seq":"5366","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:33.367346+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:33.367346+0000","last_peered":"2021-01-15T18:44:33.367346+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:33.367346+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5246,"ondisk_log_size":5246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34779243,"num_objects":4247,"num_object_clones":0,"num_object_copies":16988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5245,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2849}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5386","reported_seq":"5478","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.781365+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:34.781365+0000","last_peered":"2021-01-15T18:44:34.781365+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:34.781365+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5386,"ondisk_log_size":5386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36093952,"num_objects":4406,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5386,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5431","reported_seq":"6023","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.638075+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:34.638075+0000","last_peered":"2021-01-15T18:44:34.638075+0000","last_clean":"2021-01-15T18:44:34.638075+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:34.638075+0000","last_undegraded":"2021-01-15T18:44:34.638075+0000","last_fullsized":"2021-01-15T18:44:34.638075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5431,"ondisk_log_size":5431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36020390,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5429,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5214","reported_seq":"5424","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.514962+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:34.514962+0000","last_peered":"2021-01-15T18:44:34.514962+0000","last_clean":"2021-01-15T18:44:34.514962+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:34.514962+0000","last_undegraded":"2021-01-15T18:44:34.514962+0000","last_fullsized":"2021-01-15T18:44:34.514962+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5214,"ondisk_log_size":5214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35324070,"num_objects":4313,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5319","reported_seq":"5431","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.767751+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:34.767751+0000","last_peered":"2021-01-15T18:44:34.767751+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:34.767751+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5319,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36094118,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5392","reported_seq":"5500","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:33.333235+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:33.333235+0000","last_peered":"2021-01-15T18:44:33.333235+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:33.333235+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5392,"ondisk_log_size":5392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36573267,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3028}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5343","reported_seq":"5455","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.811957+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:34.811957+0000","last_peered":"2021-01-15T18:44:34.811957+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:34.811957+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5343,"ondisk_log_size":5343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35926099,"num_objects":4386,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2983}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5355","reported_seq":"8296","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.647625+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:34.647625+0000","last_peered":"2021-01-15T18:44:34.647625+0000","last_clean":"2021-01-15T18:44:34.647625+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:34.647625+0000","last_undegraded":"2021-01-15T18:44:34.647625+0000","last_fullsized":"2021-01-15T18:44:34.647625+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5355,"ondisk_log_size":5355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36065529,"num_objects":4404,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5257","reported_seq":"5385","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.637309+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:34.637309+0000","last_peered":"2021-01-15T18:44:34.637309+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:34.637309+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5257,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35201190,"num_objects":4298,"num_object_clones":0,"num_object_copies":17192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2950}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5252","reported_seq":"5787","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.296718+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:34.296718+0000","last_peered":"2021-01-15T18:44:34.296718+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:34.296718+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5252,"ondisk_log_size":5252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35348646,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5250,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1867776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5230","reported_seq":"8125","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:34.240301+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:34.240301+0000","last_peered":"2021-01-15T18:44:34.240301+0000","last_clean":"2021-01-15T18:44:34.240301+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:34.240301+0000","last_undegraded":"2021-01-15T18:44:34.240301+0000","last_fullsized":"2021-01-15T18:44:34.240301+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34775206,"num_objects":4246,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5228,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5437","reported_seq":"5526","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.241501+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:34.241501+0000","last_peered":"2021-01-15T18:44:34.241501+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:34.241501+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5437,"ondisk_log_size":5437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36446374,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5435,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5541","reported_seq":"5673","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:34.768392+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:34.768392+0000","last_peered":"2021-01-15T18:44:34.768392+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:34.768392+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5541,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36872524,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5538,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3051}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5225","reported_seq":"8089","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:33.518132+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:33.518132+0000","last_peered":"2021-01-15T18:44:33.518132+0000","last_clean":"2021-01-15T18:44:33.518132+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:33.518132+0000","last_undegraded":"2021-01-15T18:44:33.518132+0000","last_fullsized":"2021-01-15T18:44:33.518132+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5225,"ondisk_log_size":5225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35340454,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570738988,"num_objects":69685,"num_object_clones":0,"num_object_copies":278740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85040,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7684,"num_bytes_recovered":60792832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1104015360,"data_stored":1104015360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85067,"ondisk_log_size":85067,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215165,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450880,"kb_used_data":279600,"kb_used_omap":0,"kb_used_meta":171263,"kb_avail":93920960,"statfs":{"total":96636764160,"available":96175063040,"internally_reserved":0,"allocated":286310400,"data_stored":286096222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":175373364},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,3,6,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.5540000000000003},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":4.8650000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:43:34 2021","interfaces":[{"interface":"back","average":{"1min":6.4989999999999997,"5min":3.7130000000000001,"15min":3.2850000000000001},"min":{"1min":0.495,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":24.073,"5min":24.073,"15min":24.073},"last":4.6589999999999998},{"interface":"front","average":{"1min":5.3570000000000002,"5min":2.8109999999999999,"15min":2.23},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":19.681999999999999,"5min":19.681999999999999,"15min":19.681999999999999},"last":2.4580000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":3.4990000000000001},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":4.7670000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439072,"kb_used_data":280216,"kb_used_omap":0,"kb_used_meta":158784,"kb_avail":93932768,"statfs":{"total":96636764160,"available":96187154432,"internally_reserved":0,"allocated":286941184,"data_stored":286727006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":162594816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.752},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:43:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":2.5760000000000001,"15min":2.3820000000000001},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.34,"5min":13.926,"15min":13.926},"last":2.452},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0070000000000001,"15min":2.048},"min":{"1min":0.51800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":8.1799999999999997,"5min":10.685,"15min":10.685},"last":2.5409999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":0.99299999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.69099999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280577,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450852,"kb_used_data":279392,"kb_used_omap":0,"kb_used_meta":171391,"kb_avail":93920988,"statfs":{"total":96636764160,"available":96175091712,"internally_reserved":0,"allocated":286097408,"data_stored":285883230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":175504436},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,4,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.0379999999999998},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.7120000000000002},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":5.3140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.4269999999999996},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":5.5279999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389996,"kb_used_data":241756,"kb_used_omap":0,"kb_used_meta":148287,"kb_avail":93981844,"statfs":{"total":96636764160,"available":96237408256,"internally_reserved":0,"allocated":247558144,"data_stored":247343966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":151845940},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":0.69499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":1.5589999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.55800000000000005}]}]}]}} 2021-01-15T18:44:40.138 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:41.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:41.501 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13587/274044 objects degraded (4.958%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":331,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68511,"data_bytes":561121580,"bytes_used":1765232640,"bytes_avail":384781824000,"bytes_total":386547056640,"degraded_objects":13587,"degraded_total":274044,"degraded_ratio":0.049579629548539651,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":193147,"recovering_keys_per_sec":0,"num_objects_recovered":388,"num_bytes_recovered":2318336,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:41.579 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:41.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:42.431 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:42.482 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2021-01-15T18:44:40.500718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561121580,"num_objects":68511,"num_object_clones":0,"num_object_copies":274044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86214,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7818,"num_bytes_recovered":61480960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86241,"ondisk_log_size":86241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1723860,"kb_used_data":1064488,"kb_used_omap":2,"kb_used_meta":659261,"kb_avail":375763500,"statfs":{"total":386547056640,"available":384781824000,"internally_reserved":0,"allocated":1090035712,"data_stored":1089179000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":675083420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,8,5,12,19,10,2,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28499968,"num_objects":-3479,"num_object_clones":0,"num_object_copies":-13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"1.f","version":"40'5291","reported_seq":"5415","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.561253+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:39.561253+0000","last_peered":"2021-01-15T18:44:39.561253+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:39.561253+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5291,"ondisk_log_size":5291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34545664,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5310","reported_seq":"8234","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:39.876242+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:39.876242+0000","last_peered":"2021-01-15T18:44:39.876242+0000","last_clean":"2021-01-15T18:44:39.876242+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:39.876242+0000","last_undegraded":"2021-01-15T18:44:39.876242+0000","last_fullsized":"2021-01-15T18:44:39.876242+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5310,"ondisk_log_size":5310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34005158,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5303","reported_seq":"5423","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:38.392015+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:38.392015+0000","last_peered":"2021-01-15T18:44:38.392015+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:38.392015+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5303,"ondisk_log_size":5303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34312299,"num_objects":4190,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5302,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2792}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5473","reported_seq":"5585","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.782914+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:39.782914+0000","last_peered":"2021-01-15T18:44:39.782914+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:39.782914+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5473,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35381248,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5473,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5505","reported_seq":"6097","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:39.743879+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:39.743879+0000","last_peered":"2021-01-15T18:44:39.743879+0000","last_clean":"2021-01-15T18:44:39.743879+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:39.743879+0000","last_undegraded":"2021-01-15T18:44:39.743879+0000","last_fullsized":"2021-01-15T18:44:39.743879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5505,"ondisk_log_size":5505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35414182,"num_objects":4324,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5503,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5282","reported_seq":"5492","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:39.829629+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:39.829629+0000","last_peered":"2021-01-15T18:44:39.829629+0000","last_clean":"2021-01-15T18:44:39.829629+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:39.829629+0000","last_undegraded":"2021-01-15T18:44:39.829629+0000","last_fullsized":"2021-01-15T18:44:39.829629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5282,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34767014,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5280,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5386","reported_seq":"5511","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.806809+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:39.806809+0000","last_peered":"2021-01-15T18:44:39.806809+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:39.806809+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5386,"ondisk_log_size":5386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35545254,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5464","reported_seq":"5576","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:38.454918+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:38.454918+0000","last_peered":"2021-01-15T18:44:38.454918+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:38.454918+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5464,"ondisk_log_size":5464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35983443,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2958}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5417","reported_seq":"5546","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.806583+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:39.806583+0000","last_peered":"2021-01-15T18:44:39.806583+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:39.806583+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5417,"ondisk_log_size":5417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35319891,"num_objects":4312,"num_object_clones":0,"num_object_copies":17248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5424","reported_seq":"8365","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:39.757550+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:39.757550+0000","last_peered":"2021-01-15T18:44:39.757550+0000","last_clean":"2021-01-15T18:44:39.757550+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:39.757550+0000","last_undegraded":"2021-01-15T18:44:39.757550+0000","last_fullsized":"2021-01-15T18:44:39.757550+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5424,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35500281,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5342","reported_seq":"5475","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.806993+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:39.806993+0000","last_peered":"2021-01-15T18:44:39.806993+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:39.806993+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5342,"ondisk_log_size":5342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34504870,"num_objects":4213,"num_object_clones":0,"num_object_copies":16852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5319","reported_seq":"6023","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.337805+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:39.337805+0000","last_peered":"2021-01-15T18:44:39.337805+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:39.337805+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5319,"ondisk_log_size":5319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34799782,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5298","reported_seq":"8192","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:39.287149+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:39.287149+0000","last_peered":"2021-01-15T18:44:39.287149+0000","last_clean":"2021-01-15T18:44:39.287149+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:39.287149+0000","last_undegraded":"2021-01-15T18:44:39.287149+0000","last_fullsized":"2021-01-15T18:44:39.287149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5298,"ondisk_log_size":5298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34218150,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5296,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5511","reported_seq":"5601","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.343326+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:39.343326+0000","last_peered":"2021-01-15T18:44:39.343326+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:39.343326+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5511,"ondisk_log_size":5511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35840166,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5509,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2945}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5614","reported_seq":"5764","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:39.570817+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:39.570817+0000","last_peered":"2021-01-15T18:44:39.570817+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:39.570817+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5614,"ondisk_log_size":5614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36274508,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5611,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5302","reported_seq":"8167","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:38.406252+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:38.406252+0000","last_peered":"2021-01-15T18:44:38.406252+0000","last_clean":"2021-01-15T18:44:38.406252+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:38.406252+0000","last_undegraded":"2021-01-15T18:44:38.406252+0000","last_fullsized":"2021-01-15T18:44:38.406252+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5302,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34709670,"num_objects":4238,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5300,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":561121580,"num_objects":68511,"num_object_clones":0,"num_object_copies":274044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86214,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7818,"num_bytes_recovered":61480960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1087954944,"data_stored":1087954944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86241,"ondisk_log_size":86241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439920,"kb_used_data":274112,"kb_used_omap":0,"kb_used_meta":165759,"kb_avail":93931920,"statfs":{"total":96636764160,"available":96186286080,"internally_reserved":0,"allocated":280690688,"data_stored":280476510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":169737268},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,6,7,2,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.677},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":11.720000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.57199999999999995},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.429}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444928,"kb_used_data":275608,"kb_used_omap":0,"kb_used_meta":169280,"kb_avail":93926912,"statfs":{"total":96636764160,"available":96181157888,"internally_reserved":0,"allocated":282222592,"data_stored":282008414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":173342720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.97099999999999997},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":1.3460000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":21.952000000000002},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":2.6539999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280578,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440284,"kb_used_data":274704,"kb_used_omap":0,"kb_used_meta":165567,"kb_avail":93931556,"statfs":{"total":96636764160,"available":96185913344,"internally_reserved":0,"allocated":281296896,"data_stored":281082718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":169540660},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":7.665},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":6.0199999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":2.7120000000000002},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":5.3140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":7.5110000000000001},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":2.8849999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398728,"kb_used_data":240064,"kb_used_omap":0,"kb_used_meta":158655,"kb_avail":93973112,"statfs":{"total":96636764160,"available":96228466688,"internally_reserved":0,"allocated":245825536,"data_stored":245611358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162462772},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,10,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":6.4569999999999999,"5min":4.0490000000000004,"15min":3.1469999999999998},"min":{"1min":0.46100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":27.341000000000001,"5min":35.850999999999999,"15min":35.850999999999999},"last":4.4260000000000002},{"interface":"front","average":{"1min":3.7000000000000002,"5min":2.3620000000000001,"15min":2.0219999999999998},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":13.576000000000001,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":3.746,"5min":2.8370000000000002,"15min":2.762},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":10.446999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.629},{"interface":"front","average":{"1min":3.6890000000000001,"5min":1.9530000000000001,"15min":1.7529999999999999},"min":{"1min":0.63500000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":19.381,"5min":19.381,"15min":19.381},"last":4.0300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:38 2021","interfaces":[{"interface":"back","average":{"1min":4.3390000000000004,"5min":3.907,"15min":4.0970000000000004},"min":{"1min":0.66000000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":19.739000000000001,"5min":19.745000000000001,"15min":19.745000000000001},"last":12.045999999999999},{"interface":"front","average":{"1min":3.0569999999999999,"5min":2.52,"15min":2.5950000000000002},"min":{"1min":0.626,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":13.478999999999999,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.66600000000000004}]}]}]}} 2021-01-15T18:44:45.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:47.036 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:47.040 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:47.064 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2021-01-15T18:44:46.511532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552339756,"num_objects":67439,"num_object_clones":0,"num_object_copies":269756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87286,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7959,"num_bytes_recovered":62341120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87313,"ondisk_log_size":87313,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1721964,"kb_used_data":1049740,"kb_used_omap":2,"kb_used_meta":672125,"kb_avail":375765396,"statfs":{"total":386547056640,"available":384783765504,"internally_reserved":0,"allocated":1074933760,"data_stored":1074077048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":688256156},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,12,16,15,0,1,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24469504,"num_objects":-2987,"num_object_clones":0,"num_object_copies":-11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012377"},"pg_stats":[{"pgid":"1.f","version":"40'5356","reported_seq":"5488","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.670732+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:44.670732+0000","last_peered":"2021-01-15T18:44:44.670732+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:44.670732+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5356,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34013184,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5368","reported_seq":"8292","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.891230+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:44.891230+0000","last_peered":"2021-01-15T18:44:44.891230+0000","last_clean":"2021-01-15T18:44:44.891230+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:44.891230+0000","last_undegraded":"2021-01-15T18:44:44.891230+0000","last_fullsized":"2021-01-15T18:44:44.891230+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5368,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33530022,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5369","reported_seq":"5491","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:43.489522+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:43.489522+0000","last_peered":"2021-01-15T18:44:43.489522+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:43.489522+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5369,"ondisk_log_size":5369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33771627,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5368,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2728}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5547","reported_seq":"5671","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.772954+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:44.772954+0000","last_peered":"2021-01-15T18:44:44.772954+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:44.772954+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5547,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34775040,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5575","reported_seq":"6167","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.802692+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:44.802692+0000","last_peered":"2021-01-15T18:44:44.802692+0000","last_clean":"2021-01-15T18:44:44.802692+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:44.802692+0000","last_undegraded":"2021-01-15T18:44:44.802692+0000","last_fullsized":"2021-01-15T18:44:44.802692+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5575,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34840742,"num_objects":4254,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5360","reported_seq":"5570","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.919863+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:44.919863+0000","last_peered":"2021-01-15T18:44:44.919863+0000","last_clean":"2021-01-15T18:44:44.919863+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:44.919863+0000","last_undegraded":"2021-01-15T18:44:44.919863+0000","last_fullsized":"2021-01-15T18:44:44.919863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5360,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34128038,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5449","reported_seq":"5587","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.670949+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:44.670949+0000","last_peered":"2021-01-15T18:44:44.670949+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:44.670949+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5449,"ondisk_log_size":5449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35029158,"num_objects":4277,"num_object_clones":0,"num_object_copies":17108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5447,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2925}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5534","reported_seq":"5648","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:43.433901+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:43.433901+0000","last_peered":"2021-01-15T18:44:43.433901+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:43.433901+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5534,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35410003,"num_objects":4323,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5533,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5484","reported_seq":"5624","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.730989+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:44.730989+0000","last_peered":"2021-01-15T18:44:44.730989+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:44.730989+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5484,"ondisk_log_size":5484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34771027,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5491","reported_seq":"8432","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.734443+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:44.734443+0000","last_peered":"2021-01-15T18:44:44.734443+0000","last_clean":"2021-01-15T18:44:44.734443+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:44.734443+0000","last_undegraded":"2021-01-15T18:44:44.734443+0000","last_fullsized":"2021-01-15T18:44:44.734443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5491,"ondisk_log_size":5491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34951417,"num_objects":4268,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5399","reported_seq":"5547","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.657406+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:44.657406+0000","last_peered":"2021-01-15T18:44:44.657406+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:44.657406+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5399,"ondisk_log_size":5399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34037926,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5393","reported_seq":"6308","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.332365+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:44.332365+0000","last_peered":"2021-01-15T18:44:44.332365+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:44.332365+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5393,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34193574,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":3416064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5357","reported_seq":"8252","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.283911+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:44.283911+0000","last_peered":"2021-01-15T18:44:44.283911+0000","last_clean":"2021-01-15T18:44:44.283911+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:44.283911+0000","last_undegraded":"2021-01-15T18:44:44.283911+0000","last_fullsized":"2021-01-15T18:44:44.283911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5357,"ondisk_log_size":5357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33734822,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5355,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5578","reported_seq":"5673","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.302570+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:44.302570+0000","last_peered":"2021-01-15T18:44:44.302570+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:44.302570+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5578,"ondisk_log_size":5578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35291302,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5576,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5676","reported_seq":"5832","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.333079+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:44.333079+0000","last_peered":"2021-01-15T18:44:44.333079+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:44.333079+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5676,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35766604,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5377","reported_seq":"8242","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:43.368600+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:43.368600+0000","last_peered":"2021-01-15T18:44:43.368600+0000","last_clean":"2021-01-15T18:44:43.368600+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:43.368600+0000","last_undegraded":"2021-01-15T18:44:43.368600+0000","last_fullsized":"2021-01-15T18:44:43.368600+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5377,"ondisk_log_size":5377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34095270,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":552339756,"num_objects":67439,"num_object_clones":0,"num_object_copies":269756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87286,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7959,"num_bytes_recovered":62341120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1072996352,"data_stored":1072996352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87313,"ondisk_log_size":87313,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446008,"kb_used_data":269860,"kb_used_omap":0,"kb_used_meta":176127,"kb_avail":93925832,"statfs":{"total":96636764160,"available":96180051968,"internally_reserved":0,"allocated":276336640,"data_stored":276122462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180354100},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,8,1,0,0,0,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":5.883},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":5.9729999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":11.720000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":7.109},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.71}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434400,"kb_used_data":270980,"kb_used_omap":0,"kb_used_meta":163392,"kb_avail":93937440,"statfs":{"total":96636764160,"available":96191938560,"internally_reserved":0,"allocated":277483520,"data_stored":277269342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":167313408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,9,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.7730000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.92900000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":6.4969999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.448},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.53700000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280579,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446336,"kb_used_data":270368,"kb_used_omap":0,"kb_used_meta":175935,"kb_avail":93925504,"statfs":{"total":96636764160,"available":96179716096,"internally_reserved":0,"allocated":276856832,"data_stored":276642654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180157492},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4,5,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":5.3780000000000001},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.0899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":5.415},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.9550000000000001},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":5.0490000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395220,"kb_used_data":238532,"kb_used_omap":0,"kb_used_meta":156671,"kb_avail":93976620,"statfs":{"total":96636764160,"available":96232058880,"internally_reserved":0,"allocated":244256768,"data_stored":244042590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160431156},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.52600000000000002},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.90800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.79900000000000004}]}]}]}} 2021-01-15T18:44:47.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:48.093 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:48.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13446/269756 objects degraded (4.985%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":338,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67439,"data_bytes":552339756,"bytes_used":1763291136,"bytes_avail":384783765504,"bytes_total":386547056640,"degraded_objects":13446,"degraded_total":269756,"degraded_ratio":0.049845045151915054,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":128891,"recovering_keys_per_sec":0,"num_objects_recovered":294,"num_bytes_recovered":1548288,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":248},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:48.168 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:48.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:49.359 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:49.388 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2021-01-15T18:44:48.512044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552339756,"num_objects":67439,"num_object_clones":0,"num_object_copies":269756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87286,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7959,"num_bytes_recovered":62341120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87313,"ondisk_log_size":87313,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1721964,"kb_used_data":1049740,"kb_used_omap":2,"kb_used_meta":672125,"kb_avail":375765396,"statfs":{"total":386547056640,"available":384783765504,"internally_reserved":0,"allocated":1074933760,"data_stored":1074077048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":688256156},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,12,16,15,0,1,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18399232,"num_objects":-2246,"num_object_clones":0,"num_object_copies":-8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012380"},"pg_stats":[{"pgid":"1.f","version":"40'5356","reported_seq":"5488","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.670732+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:44.670732+0000","last_peered":"2021-01-15T18:44:44.670732+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:44.670732+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5356,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34013184,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5368","reported_seq":"8292","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.891230+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:44.891230+0000","last_peered":"2021-01-15T18:44:44.891230+0000","last_clean":"2021-01-15T18:44:44.891230+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:44.891230+0000","last_undegraded":"2021-01-15T18:44:44.891230+0000","last_fullsized":"2021-01-15T18:44:44.891230+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5368,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33530022,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5369","reported_seq":"5491","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:43.489522+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:43.489522+0000","last_peered":"2021-01-15T18:44:43.489522+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:43.489522+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5369,"ondisk_log_size":5369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33771627,"num_objects":4124,"num_object_clones":0,"num_object_copies":16496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5368,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2728}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5547","reported_seq":"5671","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.772954+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:44.772954+0000","last_peered":"2021-01-15T18:44:44.772954+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:44.772954+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5547,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34775040,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2850}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5575","reported_seq":"6167","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.802692+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:44.802692+0000","last_peered":"2021-01-15T18:44:44.802692+0000","last_clean":"2021-01-15T18:44:44.802692+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:44.802692+0000","last_undegraded":"2021-01-15T18:44:44.802692+0000","last_fullsized":"2021-01-15T18:44:44.802692+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5575,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34840742,"num_objects":4254,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5360","reported_seq":"5570","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.919863+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:44.919863+0000","last_peered":"2021-01-15T18:44:44.919863+0000","last_clean":"2021-01-15T18:44:44.919863+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:44.919863+0000","last_undegraded":"2021-01-15T18:44:44.919863+0000","last_fullsized":"2021-01-15T18:44:44.919863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5360,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34128038,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5449","reported_seq":"5587","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.670949+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:44.670949+0000","last_peered":"2021-01-15T18:44:44.670949+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:44.670949+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5449,"ondisk_log_size":5449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35029158,"num_objects":4277,"num_object_clones":0,"num_object_copies":17108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5447,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2925}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5534","reported_seq":"5648","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:43.433901+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:43.433901+0000","last_peered":"2021-01-15T18:44:43.433901+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:43.433901+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5534,"ondisk_log_size":5534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35410003,"num_objects":4323,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5533,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5484","reported_seq":"5624","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.730989+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:44.730989+0000","last_peered":"2021-01-15T18:44:44.730989+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:44.730989+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5484,"ondisk_log_size":5484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34771027,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5491","reported_seq":"8432","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.734443+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:44.734443+0000","last_peered":"2021-01-15T18:44:44.734443+0000","last_clean":"2021-01-15T18:44:44.734443+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:44.734443+0000","last_undegraded":"2021-01-15T18:44:44.734443+0000","last_fullsized":"2021-01-15T18:44:44.734443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5491,"ondisk_log_size":5491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34951417,"num_objects":4268,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5399","reported_seq":"5547","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.657406+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:44.657406+0000","last_peered":"2021-01-15T18:44:44.657406+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:44.657406+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5399,"ondisk_log_size":5399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34037926,"num_objects":4156,"num_object_clones":0,"num_object_copies":16624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5393","reported_seq":"6308","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.332365+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:44.332365+0000","last_peered":"2021-01-15T18:44:44.332365+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:44.332365+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5393,"ondisk_log_size":5393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34193574,"num_objects":4175,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":3416064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5357","reported_seq":"8252","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:44.283911+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:44.283911+0000","last_peered":"2021-01-15T18:44:44.283911+0000","last_clean":"2021-01-15T18:44:44.283911+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:44.283911+0000","last_undegraded":"2021-01-15T18:44:44.283911+0000","last_fullsized":"2021-01-15T18:44:44.283911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5357,"ondisk_log_size":5357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33734822,"num_objects":4119,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5355,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5578","reported_seq":"5673","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.302570+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:44.302570+0000","last_peered":"2021-01-15T18:44:44.302570+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:44.302570+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5578,"ondisk_log_size":5578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35291302,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5576,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5676","reported_seq":"5832","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:44.333079+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:44.333079+0000","last_peered":"2021-01-15T18:44:44.333079+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:44.333079+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5676,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35766604,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5377","reported_seq":"8242","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:43.368600+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:43.368600+0000","last_peered":"2021-01-15T18:44:43.368600+0000","last_clean":"2021-01-15T18:44:43.368600+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:43.368600+0000","last_undegraded":"2021-01-15T18:44:43.368600+0000","last_fullsized":"2021-01-15T18:44:43.368600+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5377,"ondisk_log_size":5377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34095270,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":552339756,"num_objects":67439,"num_object_clones":0,"num_object_copies":269756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87286,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7959,"num_bytes_recovered":62341120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1072996352,"data_stored":1072996352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87313,"ondisk_log_size":87313,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446008,"kb_used_data":269860,"kb_used_omap":0,"kb_used_meta":176127,"kb_avail":93925832,"statfs":{"total":96636764160,"available":96180051968,"internally_reserved":0,"allocated":276336640,"data_stored":276122462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180354100},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,8,1,0,0,0,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":5.883},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":5.9729999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":11.720000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":7.109},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.71}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434400,"kb_used_data":270980,"kb_used_omap":0,"kb_used_meta":163392,"kb_avail":93937440,"statfs":{"total":96636764160,"available":96191938560,"internally_reserved":0,"allocated":277483520,"data_stored":277269342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":167313408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,9,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.7730000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":0.92900000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":6.4969999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":1.448},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.53700000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280579,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446336,"kb_used_data":270368,"kb_used_omap":0,"kb_used_meta":175935,"kb_avail":93925504,"statfs":{"total":96636764160,"available":96179716096,"internally_reserved":0,"allocated":276856832,"data_stored":276642654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180157492},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4,5,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":5.3780000000000001},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.0899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:43:48 2021","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":2.8660000000000001,"15min":2.4350000000000001},"min":{"1min":0.55800000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":10.853,"5min":27.77,"15min":27.77},"last":5.415},{"interface":"front","average":{"1min":2.7599999999999998,"5min":3.1499999999999999,"15min":2.5979999999999999},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.6430000000000007,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.9550000000000001},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":5.0490000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395220,"kb_used_data":238532,"kb_used_omap":0,"kb_used_meta":156671,"kb_avail":93976620,"statfs":{"total":96636764160,"available":96232058880,"internally_reserved":0,"allocated":244256768,"data_stored":244042590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160431156},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.52600000000000002},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.90800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.79900000000000004}]}]}]}} 2021-01-15T18:44:52.391 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:53.295 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:53.321 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:53.366 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2021-01-15T18:44:52.516310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543123756,"num_objects":66314,"num_object_clones":0,"num_object_copies":265256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":88411,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8075,"num_bytes_recovered":63062016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88438,"ondisk_log_size":88438,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1715392,"kb_used_data":1034684,"kb_used_omap":2,"kb_used_meta":680573,"kb_avail":375771968,"statfs":{"total":386547056640,"available":384790495232,"internally_reserved":0,"allocated":1059516416,"data_stored":1058659704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":696906908},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,7,12,19,7,0,0,0,2,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":100,"apply_latency_ms":100,"commit_latency_ns":100000000,"apply_latency_ns":100000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.015591"},"pg_stats":[{"pgid":"1.f","version":"40'5421","reported_seq":"5565","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.748198+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:49.748198+0000","last_peered":"2021-01-15T18:44:49.748198+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:49.748198+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5421,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5445","reported_seq":"8369","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.823769+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:49.823769+0000","last_peered":"2021-01-15T18:44:49.823769+0000","last_clean":"2021-01-15T18:44:49.823769+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:49.823769+0000","last_undegraded":"2021-01-15T18:44:49.823769+0000","last_fullsized":"2021-01-15T18:44:49.823769+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5445,"ondisk_log_size":5445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32899238,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5439","reported_seq":"5563","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:48.389330+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:48.389330+0000","last_peered":"2021-01-15T18:44:48.389330+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:48.389330+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5439,"ondisk_log_size":5439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33198187,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5438,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2658}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5625","reported_seq":"5761","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.802630+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:49.802630+0000","last_peered":"2021-01-15T18:44:49.802630+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:49.802630+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5625,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34136064,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5637","reported_seq":"6229","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.825024+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:49.825024+0000","last_peered":"2021-01-15T18:44:49.825024+0000","last_clean":"2021-01-15T18:44:49.825024+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:49.825024+0000","last_undegraded":"2021-01-15T18:44:49.825024+0000","last_fullsized":"2021-01-15T18:44:49.825024+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5637,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34332838,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5443","reported_seq":"5654","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.820100+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:49.820100+0000","last_peered":"2021-01-15T18:44:49.820100+0000","last_clean":"2021-01-15T18:44:49.820100+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:49.820100+0000","last_undegraded":"2021-01-15T18:44:49.820100+0000","last_fullsized":"2021-01-15T18:44:49.820100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5443,"ondisk_log_size":5443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33448102,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5508","reported_seq":"5650","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.443373+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:49.443373+0000","last_peered":"2021-01-15T18:44:49.443373+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:49.443373+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5508,"ondisk_log_size":5508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34545830,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5506,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5609","reported_seq":"5724","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:48.464347+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:48.464347+0000","last_peered":"2021-01-15T18:44:48.464347+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:48.464347+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5609,"ondisk_log_size":5609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34795603,"num_objects":4248,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5561","reported_seq":"5711","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.752452+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:49.752452+0000","last_peered":"2021-01-15T18:44:49.752452+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:49.752452+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5561,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34140243,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2784}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5560","reported_seq":"8501","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.688198+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:49.688198+0000","last_peered":"2021-01-15T18:44:49.688198+0000","last_clean":"2021-01-15T18:44:49.688198+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:49.688198+0000","last_undegraded":"2021-01-15T18:44:49.688198+0000","last_fullsized":"2021-01-15T18:44:49.688198+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5560,"ondisk_log_size":5560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34386169,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5458","reported_seq":"5610","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.709260+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:49.709260+0000","last_peered":"2021-01-15T18:44:49.709260+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:49.709260+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5458,"ondisk_log_size":5458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33554598,"num_objects":4097,"num_object_clones":0,"num_object_copies":16388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5456","reported_seq":"6549","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.354014+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:49.354014+0000","last_peered":"2021-01-15T18:44:49.354014+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:49.354014+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5456,"ondisk_log_size":5456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33677478,"num_objects":4112,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5454,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":4136960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3252}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5421","reported_seq":"8316","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.221167+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:49.221167+0000","last_peered":"2021-01-15T18:44:49.221167+0000","last_clean":"2021-01-15T18:44:49.221167+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:49.221167+0000","last_undegraded":"2021-01-15T18:44:49.221167+0000","last_fullsized":"2021-01-15T18:44:49.221167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5421,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33210534,"num_objects":4055,"num_object_clones":0,"num_object_copies":16220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5657","reported_seq":"5756","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.324452+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:49.324452+0000","last_peered":"2021-01-15T18:44:49.324452+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:49.324452+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5657,"ondisk_log_size":5657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34644134,"num_objects":4230,"num_object_clones":0,"num_object_copies":16920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5759","reported_seq":"5919","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.769330+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:49.769330+0000","last_peered":"2021-01-15T18:44:49.769330+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:49.769330+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5759,"ondisk_log_size":5759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35086668,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5439","reported_seq":"8303","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:48.437288+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:48.437288+0000","last_peered":"2021-01-15T18:44:48.437288+0000","last_clean":"2021-01-15T18:44:48.437288+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:48.437288+0000","last_undegraded":"2021-01-15T18:44:48.437288+0000","last_fullsized":"2021-01-15T18:44:48.437288+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5439,"ondisk_log_size":5439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33587366,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5437,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":543123756,"num_objects":66314,"num_object_clones":0,"num_object_copies":265256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":88411,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8075,"num_bytes_recovered":63062016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057632256,"data_stored":1057632256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88438,"ondisk_log_size":88438,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435448,"kb_used_data":265232,"kb_used_omap":0,"kb_used_meta":170175,"kb_avail":93936392,"statfs":{"total":96636764160,"available":96190865408,"internally_reserved":0,"allocated":271597568,"data_stored":271383390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":174259252},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":6.359}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":6.5810000000000004},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":6.6829999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.92100000000000004},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.4720000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444460,"kb_used_data":266500,"kb_used_omap":0,"kb_used_meta":177920,"kb_avail":93927380,"statfs":{"total":96636764160,"available":96181637120,"internally_reserved":0,"allocated":272896000,"data_stored":272681822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":182190080},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,0,6,2,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:43:49 2021","interfaces":[{"interface":"back","average":{"1min":3.3999999999999999,"5min":3.5470000000000002,"15min":3.6850000000000001},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.032999999999999,"5min":36.380000000000003,"15min":36.380000000000003},"last":1.925},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.1219999999999999,"15min":1.3640000000000001},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":19.579000000000001,"5min":21.344999999999999,"15min":21.344999999999999},"last":8.2390000000000008}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.1279999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":2.2320000000000002},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":0.81799999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435796,"kb_used_data":265708,"kb_used_omap":0,"kb_used_meta":170047,"kb_avail":93936044,"statfs":{"total":96636764160,"available":96190509056,"internally_reserved":0,"allocated":272084992,"data_stored":271870814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":174128180},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.6099999999999999},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.1720000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.4669999999999996},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.2370000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":237244,"kb_used_omap":0,"kb_used_meta":162431,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":242937856,"data_stored":242723678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166329396},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.3129999999999997},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.6040000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.93400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":10.281000000000001},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":9.7089999999999996}]}]}]}} 2021-01-15T18:44:53.368 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:44:54.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:54.311 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13332/265256 objects degraded (5.026%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":344,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66314,"data_bytes":543123756,"bytes_used":1756561408,"bytes_avail":384790495232,"bytes_total":386547056640,"degraded_objects":13332,"degraded_total":265256,"degraded_ratio":0.050260880055493562,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":131583,"recovering_keys_per_sec":0,"num_objects_recovered":257,"num_bytes_recovered":1581056,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":182},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:44:54.418 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:44:54.419 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:55.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:55.421 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:55.457 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2021-01-15T18:44:54.518830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539863340,"num_objects":65916,"num_object_clones":0,"num_object_copies":263664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":88809,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8177,"num_bytes_recovered":63848448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88836,"ondisk_log_size":88836,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1715220,"kb_used_data":1026064,"kb_used_omap":2,"kb_used_meta":689021,"kb_avail":375772140,"statfs":{"total":386547056640,"available":384790671360,"internally_reserved":0,"allocated":1050689536,"data_stored":1049832824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":705557660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,5,15,17,5,0,0,0,2,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":100,"apply_latency_ms":100,"commit_latency_ns":100000000,"apply_latency_ns":100000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21258240,"num_objects":-2595,"num_object_clones":0,"num_object_copies":-10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":359,"num_bytes_recovered":2367488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013998"},"pg_stats":[{"pgid":"1.f","version":"40'5421","reported_seq":"5565","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.748198+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:49.748198+0000","last_peered":"2021-01-15T18:44:49.748198+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:49.748198+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5421,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33480704,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5445","reported_seq":"8369","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.823769+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:49.823769+0000","last_peered":"2021-01-15T18:44:49.823769+0000","last_clean":"2021-01-15T18:44:49.823769+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:49.823769+0000","last_undegraded":"2021-01-15T18:44:49.823769+0000","last_fullsized":"2021-01-15T18:44:49.823769+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5445,"ondisk_log_size":5445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32899238,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5509","reported_seq":"5635","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:53.068798+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:53.068798+0000","last_peered":"2021-01-15T18:44:53.068798+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:53.068798+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5509,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32624747,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5508,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5625","reported_seq":"5761","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.802630+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:49.802630+0000","last_peered":"2021-01-15T18:44:49.802630+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:49.802630+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5625,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34136064,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5637","reported_seq":"6229","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.825024+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:49.825024+0000","last_peered":"2021-01-15T18:44:49.825024+0000","last_clean":"2021-01-15T18:44:49.825024+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:49.825024+0000","last_undegraded":"2021-01-15T18:44:49.825024+0000","last_fullsized":"2021-01-15T18:44:49.825024+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5637,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34332838,"num_objects":4192,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5443","reported_seq":"5654","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.820100+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:49.820100+0000","last_peered":"2021-01-15T18:44:49.820100+0000","last_clean":"2021-01-15T18:44:49.820100+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:49.820100+0000","last_undegraded":"2021-01-15T18:44:49.820100+0000","last_fullsized":"2021-01-15T18:44:49.820100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5443,"ondisk_log_size":5443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33448102,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5508","reported_seq":"5650","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.443373+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:49.443373+0000","last_peered":"2021-01-15T18:44:49.443373+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:49.443373+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5508,"ondisk_log_size":5508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34545830,"num_objects":4218,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5506,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2868}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5672","reported_seq":"5798","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:53.345425+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:53.345425+0000","last_peered":"2021-01-15T18:44:53.345425+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:53.345425+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5672,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34279507,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5561","reported_seq":"5711","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.752452+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:49.752452+0000","last_peered":"2021-01-15T18:44:49.752452+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:49.752452+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5561,"ondisk_log_size":5561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34140243,"num_objects":4168,"num_object_clones":0,"num_object_copies":16672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2784}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5560","reported_seq":"8501","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:49.688198+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:49.688198+0000","last_peered":"2021-01-15T18:44:49.688198+0000","last_clean":"2021-01-15T18:44:49.688198+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:49.688198+0000","last_undegraded":"2021-01-15T18:44:49.688198+0000","last_fullsized":"2021-01-15T18:44:49.688198+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5560,"ondisk_log_size":5560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34386169,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5458","reported_seq":"5610","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.709260+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:49.709260+0000","last_peered":"2021-01-15T18:44:49.709260+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:49.709260+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5458,"ondisk_log_size":5458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33554598,"num_objects":4097,"num_object_clones":0,"num_object_copies":16388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5523","reported_seq":"6810","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.294415+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:54.294415+0000","last_peered":"2021-01-15T18:44:54.294415+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:54.294415+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5523,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33128614,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5521,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":4923392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5482","reported_seq":"8377","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.018684+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:54.018684+0000","last_peered":"2021-01-15T18:44:54.018684+0000","last_clean":"2021-01-15T18:44:54.018684+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:54.018684+0000","last_undegraded":"2021-01-15T18:44:54.018684+0000","last_fullsized":"2021-01-15T18:44:54.018684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5482,"ondisk_log_size":5482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32710822,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5480,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5724","reported_seq":"5823","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.292907+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:54.292907+0000","last_peered":"2021-01-15T18:44:54.292907+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:54.292907+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5724,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34095270,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5759","reported_seq":"5919","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:49.769330+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:49.769330+0000","last_peered":"2021-01-15T18:44:49.769330+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:49.769330+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5759,"ondisk_log_size":5759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35086668,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2847}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5509","reported_seq":"8374","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:53.486061+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:53.486061+0000","last_peered":"2021-01-15T18:44:53.486061+0000","last_clean":"2021-01-15T18:44:53.486061+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:53.486061+0000","last_undegraded":"2021-01-15T18:44:53.486061+0000","last_fullsized":"2021-01-15T18:44:53.486061+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5509,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33013926,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5507,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539863340,"num_objects":65916,"num_object_clones":0,"num_object_copies":263664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":88809,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8177,"num_bytes_recovered":63848448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048780800,"data_stored":1048780800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88836,"ondisk_log_size":88836,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435448,"kb_used_data":265232,"kb_used_omap":0,"kb_used_meta":170175,"kb_avail":93936392,"statfs":{"total":96636764160,"available":96190865408,"internally_reserved":0,"allocated":271597568,"data_stored":271383390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":174259252},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,6,1,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":6.359}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":6.5810000000000004},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":6.6829999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.92100000000000004},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":6.4720000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434184,"kb_used_data":262240,"kb_used_omap":0,"kb_used_meta":171904,"kb_avail":93937656,"statfs":{"total":96636764160,"available":96192159744,"internally_reserved":0,"allocated":268533760,"data_stored":268319582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":176029696},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,4,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5129999999999999},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":1.6140000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.76},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":1.8520000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":9.8520000000000003},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.5049999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445900,"kb_used_data":261348,"kb_used_omap":0,"kb_used_meta":184511,"kb_avail":93925940,"statfs":{"total":96636764160,"available":96180162560,"internally_reserved":0,"allocated":267620352,"data_stored":267406174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":188939316},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.6099999999999999},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.1720000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.4669999999999996},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.2370000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399688,"kb_used_data":237244,"kb_used_omap":0,"kb_used_meta":162431,"kb_avail":93972152,"statfs":{"total":96636764160,"available":96227483648,"internally_reserved":0,"allocated":242937856,"data_stored":242723678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166329396},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.3129999999999997},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.6040000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.93400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":10.281000000000001},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":9.7089999999999996}]}]}]}} 2021-01-15T18:44:58.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:44:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:44:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:44:59.576 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2021-01-15T18:44:58.521810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534341932,"num_objects":65242,"num_object_clones":0,"num_object_copies":260968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89483,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8221,"num_bytes_recovered":63848448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89510,"ondisk_log_size":89510,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1717888,"kb_used_data":1020376,"kb_used_omap":2,"kb_used_meta":697341,"kb_avail":375769472,"statfs":{"total":386547056640,"available":384787939328,"internally_reserved":0,"allocated":1044865024,"data_stored":1044008312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":714077340},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,5,11,13,8,0,0,0,2,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":94,"apply_latency_ms":94,"commit_latency_ns":94000000,"apply_latency_ns":94000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010277"},"pg_stats":[{"pgid":"1.f","version":"40'5487","reported_seq":"5647","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.771621+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:54.771621+0000","last_peered":"2021-01-15T18:44:54.771621+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:54.771621+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5487,"ondisk_log_size":5487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5487,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5507","reported_seq":"8431","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.770587+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:54.770587+0000","last_peered":"2021-01-15T18:44:54.770587+0000","last_clean":"2021-01-15T18:44:54.770587+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:54.770587+0000","last_undegraded":"2021-01-15T18:44:54.770587+0000","last_fullsized":"2021-01-15T18:44:54.770587+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5507,"ondisk_log_size":5507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32391334,"num_objects":3955,"num_object_clones":0,"num_object_copies":15820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5509","reported_seq":"5635","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:53.068798+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:53.068798+0000","last_peered":"2021-01-15T18:44:53.068798+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:53.068798+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5509,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32624747,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5508,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5693","reported_seq":"5845","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.819040+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:54.819040+0000","last_peered":"2021-01-15T18:44:54.819040+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:54.819040+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5693,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33579008,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5693,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5696","reported_seq":"6288","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.903517+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:54.903517+0000","last_peered":"2021-01-15T18:44:54.903517+0000","last_clean":"2021-01-15T18:44:54.903517+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:54.903517+0000","last_undegraded":"2021-01-15T18:44:54.903517+0000","last_fullsized":"2021-01-15T18:44:54.903517+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5696,"ondisk_log_size":5696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33849510,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5508","reported_seq":"5719","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.824243+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:54.824243+0000","last_peered":"2021-01-15T18:44:54.824243+0000","last_clean":"2021-01-15T18:44:54.824243+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:54.824243+0000","last_undegraded":"2021-01-15T18:44:54.824243+0000","last_fullsized":"2021-01-15T18:44:54.824243+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5508,"ondisk_log_size":5508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32915622,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5506,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5591","reported_seq":"5749","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.746482+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:54.746482+0000","last_peered":"2021-01-15T18:44:54.746482+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:54.746482+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5591,"ondisk_log_size":5591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33865894,"num_objects":4135,"num_object_clones":0,"num_object_copies":16540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5589,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5672","reported_seq":"5798","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:53.345425+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:53.345425+0000","last_peered":"2021-01-15T18:44:53.345425+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:53.345425+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5672,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34279507,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5634","reported_seq":"5796","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.773512+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:54.773512+0000","last_peered":"2021-01-15T18:44:54.773512+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:54.773512+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5634,"ondisk_log_size":5634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33542227,"num_objects":4095,"num_object_clones":0,"num_object_copies":16380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5628","reported_seq":"8569","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.706184+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:54.706184+0000","last_peered":"2021-01-15T18:44:54.706184+0000","last_clean":"2021-01-15T18:44:54.706184+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:54.706184+0000","last_undegraded":"2021-01-15T18:44:54.706184+0000","last_fullsized":"2021-01-15T18:44:54.706184+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5628,"ondisk_log_size":5628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33829113,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5522","reported_seq":"5688","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.633904+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:54.633904+0000","last_peered":"2021-01-15T18:44:54.633904+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:54.633904+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5522,"ondisk_log_size":5522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33030310,"num_objects":4033,"num_object_clones":0,"num_object_copies":16132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5520,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5523","reported_seq":"6810","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.294415+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:54.294415+0000","last_peered":"2021-01-15T18:44:54.294415+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:54.294415+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5523,"ondisk_log_size":5523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33128614,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5521,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":4923392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5482","reported_seq":"8377","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:54.018684+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:54.018684+0000","last_peered":"2021-01-15T18:44:54.018684+0000","last_clean":"2021-01-15T18:44:54.018684+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:54.018684+0000","last_undegraded":"2021-01-15T18:44:54.018684+0000","last_fullsized":"2021-01-15T18:44:54.018684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5482,"ondisk_log_size":5482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32710822,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5480,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5724","reported_seq":"5823","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.292907+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:54.292907+0000","last_peered":"2021-01-15T18:44:54.292907+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:54.292907+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5724,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34095270,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5825","reported_seq":"5997","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:54.789109+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:54.789109+0000","last_peered":"2021-01-15T18:44:54.789109+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:54.789109+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5825,"ondisk_log_size":5825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34545996,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2787}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5509","reported_seq":"8374","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:53.486061+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:53.486061+0000","last_peered":"2021-01-15T18:44:53.486061+0000","last_clean":"2021-01-15T18:44:53.486061+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:53.486061+0000","last_undegraded":"2021-01-15T18:44:53.486061+0000","last_fullsized":"2021-01-15T18:44:53.486061+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5509,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33013926,"num_objects":4031,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5507,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534341932,"num_objects":65242,"num_object_clones":0,"num_object_copies":260968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89483,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8221,"num_bytes_recovered":63848448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1042960384,"data_stored":1042960384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89510,"ondisk_log_size":89510,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215169,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441552,"kb_used_data":260948,"kb_used_omap":0,"kb_used_meta":180543,"kb_avail":93930288,"statfs":{"total":96636764160,"available":96184614912,"internally_reserved":0,"allocated":267210752,"data_stored":266996574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":184876084},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":26.533999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":26.635000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":0.66900000000000004},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.75800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":434184,"kb_used_data":262240,"kb_used_omap":0,"kb_used_meta":171904,"kb_avail":93937656,"statfs":{"total":96636764160,"available":96192159744,"internally_reserved":0,"allocated":268533760,"data_stored":268319582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":176029696},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,4,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":2.5129999999999999},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":1.6140000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.76},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":1.8520000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:43:54 2021","interfaces":[{"interface":"back","average":{"1min":3.105,"5min":2.665,"15min":2.415},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":14.382,"5min":21.917000000000002,"15min":21.917000000000002},"last":9.8520000000000003},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.04,"15min":2.528},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":13.097,"5min":13.901,"15min":13.901},"last":1.5049999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":445900,"kb_used_data":261348,"kb_used_omap":0,"kb_used_meta":184511,"kb_avail":93925940,"statfs":{"total":96636764160,"available":96180162560,"internally_reserved":0,"allocated":267620352,"data_stored":267406174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":188939316},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.6099999999999999},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.1720000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:43:57 2021","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":3.5630000000000002,"15min":4.3170000000000002},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":6.8339999999999996,"5min":41.984999999999999,"15min":41.984999999999999},"last":6.4669999999999996},{"interface":"front","average":{"1min":8.3819999999999997,"5min":4.5499999999999998,"15min":2.9369999999999998},"min":{"1min":0.78000000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":56.927,"5min":56.927,"15min":56.927},"last":1.2370000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396252,"kb_used_data":235840,"kb_used_omap":0,"kb_used_meta":160383,"kb_avail":93975588,"statfs":{"total":96636764160,"available":96231002112,"internally_reserved":0,"allocated":241500160,"data_stored":241285982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164232244},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.3129999999999997},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.6040000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.93400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":10.281000000000001},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":9.7089999999999996}]}]}]}} 2021-01-15T18:44:59.578 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:00.878 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:00.878 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13041/256380 objects degraded (5.087%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":351,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":64095,"data_bytes":524945708,"bytes_used":1740308480,"bytes_avail":384806748160,"bytes_total":386547056640,"degraded_objects":13041,"degraded_total":256380,"degraded_ratio":0.050865902176456819,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":186891,"recovering_keys_per_sec":0,"num_objects_recovered":407,"num_bytes_recovered":2244608,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":278},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2021-01-15T18:44:14.485067+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:45:00.961 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:00.962 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:01.909 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:01.960 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2021-01-15T18:45:00.522283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":524945708,"num_objects":64095,"num_object_clones":0,"num_object_copies":256380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64095,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90630,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8366,"num_bytes_recovered":64585728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90657,"ondisk_log_size":90657,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1699520,"kb_used_data":1004888,"kb_used_omap":2,"kb_used_meta":694525,"kb_avail":375787840,"statfs":{"total":386547056640,"available":384806748160,"internally_reserved":0,"allocated":1029005312,"data_stored":1028148600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":711193756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,14,21,14,0,0,5,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":97,"apply_latency_ms":97,"commit_latency_ns":97000000,"apply_latency_ns":97000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27394048,"num_objects":-3344,"num_object_clones":0,"num_object_copies":-13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3344,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":2244608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010238"},"pg_stats":[{"pgid":"1.f","version":"40'5556","reported_seq":"5740","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.841678+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:59.841678+0000","last_peered":"2021-01-15T18:44:59.841678+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:59.841678+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5556,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32374784,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5556,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2665}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5581","reported_seq":"8505","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.853476+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:59.853476+0000","last_peered":"2021-01-15T18:44:59.853476+0000","last_clean":"2021-01-15T18:44:59.853476+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:59.853476+0000","last_undegraded":"2021-01-15T18:44:59.853476+0000","last_fullsized":"2021-01-15T18:44:59.853476+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5581,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31785126,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5573","reported_seq":"5700","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:58.516496+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:44:58.516496+0000","last_peered":"2021-01-15T18:44:58.516496+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:44:58.516496+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5573,"ondisk_log_size":5573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32100459,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5572,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5766","reported_seq":"5938","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.833826+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:59.833826+0000","last_peered":"2021-01-15T18:44:59.833826+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:59.833826+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5766,"ondisk_log_size":5766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32980992,"num_objects":4026,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5776","reported_seq":"6368","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.831278+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:59.831278+0000","last_peered":"2021-01-15T18:44:59.831278+0000","last_clean":"2021-01-15T18:44:59.831278+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:59.831278+0000","last_undegraded":"2021-01-15T18:44:59.831278+0000","last_fullsized":"2021-01-15T18:44:59.831278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5776,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33194150,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5579","reported_seq":"5790","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.792696+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:59.792696+0000","last_peered":"2021-01-15T18:44:59.792696+0000","last_clean":"2021-01-15T18:44:59.792696+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:59.792696+0000","last_undegraded":"2021-01-15T18:44:59.792696+0000","last_fullsized":"2021-01-15T18:44:59.792696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5579,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32333990,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5665","reported_seq":"5843","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.770023+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:59.770023+0000","last_peered":"2021-01-15T18:44:59.770023+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:59.770023+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5665,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33259686,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5753","reported_seq":"5879","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:58.256270+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:44:58.256270+0000","last_peered":"2021-01-15T18:44:58.256270+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:44:58.256270+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5753,"ondisk_log_size":5753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33615955,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5752,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5704","reported_seq":"5886","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.792501+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:59.792501+0000","last_peered":"2021-01-15T18:44:59.792501+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:59.792501+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5704,"ondisk_log_size":5704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32968787,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5701","reported_seq":"8642","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.688178+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:59.688178+0000","last_peered":"2021-01-15T18:44:59.688178+0000","last_clean":"2021-01-15T18:44:59.688178+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:59.688178+0000","last_undegraded":"2021-01-15T18:44:59.688178+0000","last_fullsized":"2021-01-15T18:44:59.688178+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5701,"ondisk_log_size":5701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33231097,"num_objects":4058,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5698,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5597","reported_seq":"5769","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.850278+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:59.850278+0000","last_peered":"2021-01-15T18:44:59.850278+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:59.850278+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5597,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32415910,"num_objects":3958,"num_object_clones":0,"num_object_copies":15832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5587","reported_seq":"7053","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.359281+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:44:59.359281+0000","last_peered":"2021-01-15T18:44:59.359281+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:44:59.359281+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5587,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32604326,"num_objects":3981,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":701,"num_bytes_recovered":5660672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5542","reported_seq":"8437","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.225205+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:44:59.225205+0000","last_peered":"2021-01-15T18:44:59.225205+0000","last_clean":"2021-01-15T18:44:59.225205+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:44:59.225205+0000","last_undegraded":"2021-01-15T18:44:59.225205+0000","last_fullsized":"2021-01-15T18:44:59.225205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5542,"ondisk_log_size":5542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32219302,"num_objects":3934,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5796","reported_seq":"5899","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.207283+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:44:59.207283+0000","last_peered":"2021-01-15T18:44:59.207283+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:44:59.207283+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5796,"ondisk_log_size":5796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33505446,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5894","reported_seq":"6080","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.631015+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:59.631015+0000","last_peered":"2021-01-15T18:44:59.631015+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:59.631015+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5894,"ondisk_log_size":5894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33980748,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5891,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5587","reported_seq":"8452","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:58.484841+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:44:58.484841+0000","last_peered":"2021-01-15T18:44:58.484841+0000","last_clean":"2021-01-15T18:44:58.484841+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:44:58.484841+0000","last_undegraded":"2021-01-15T18:44:58.484841+0000","last_fullsized":"2021-01-15T18:44:58.484841+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5587,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32374950,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":524945708,"num_objects":64095,"num_object_clones":0,"num_object_copies":256380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64095,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90630,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8366,"num_bytes_recovered":64585728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027346432,"data_stored":1027346432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90657,"ondisk_log_size":90657,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435172,"kb_used_data":256388,"kb_used_omap":0,"kb_used_meta":178751,"kb_avail":93936668,"statfs":{"total":96636764160,"available":96191148032,"internally_reserved":0,"allocated":262541312,"data_stored":262327134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183041076},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.0269999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.125},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.212}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":1.833},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":3.3149999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440116,"kb_used_data":257772,"kb_used_omap":0,"kb_used_meta":182336,"kb_avail":93931724,"statfs":{"total":96636764160,"available":96186085376,"internally_reserved":0,"allocated":263958528,"data_stored":263744350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":186712064},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":27.870000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":15.721}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":15.609},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":16.904},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":15.856999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280582,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435500,"kb_used_data":256832,"kb_used_omap":0,"kb_used_meta":178623,"kb_avail":93936340,"statfs":{"total":96636764160,"available":96190812160,"internally_reserved":0,"allocated":262995968,"data_stored":262781790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":182910004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,6,4,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":2.2719999999999998},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":3.4870000000000001},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":3.601}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.86299999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388732,"kb_used_data":233896,"kb_used_omap":0,"kb_used_meta":154815,"kb_avail":93983108,"statfs":{"total":96636764160,"available":96238702592,"internally_reserved":0,"allocated":239509504,"data_stored":239295326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":158530612},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.7579999999999991},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.843}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.755},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":7.9740000000000002}]}]}]}} 2021-01-15T18:45:04.962 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:05.972 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:05.994 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:06.025 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2021-01-15T18:45:04.526627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":521398572,"num_objects":63662,"num_object_clones":0,"num_object_copies":254648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91063,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8451,"num_bytes_recovered":65282048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91090,"ondisk_log_size":91090,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1694776,"kb_used_data":995592,"kb_used_omap":2,"kb_used_meta":699069,"kb_avail":375792584,"statfs":{"total":386547056640,"available":384811606016,"internally_reserved":0,"allocated":1019486208,"data_stored":1018629496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":715846812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,13,23,14,0,0,5,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21725184,"num_objects":-2652,"num_object_clones":0,"num_object_copies":-10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2652,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":376,"num_bytes_recovered":2220032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010317"},"pg_stats":[{"pgid":"1.f","version":"40'5556","reported_seq":"5740","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.841678+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:44:59.841678+0000","last_peered":"2021-01-15T18:44:59.841678+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:44:59.841678+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5556,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32374784,"num_objects":3952,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5556,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2665}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5581","reported_seq":"8505","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.853476+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:44:59.853476+0000","last_peered":"2021-01-15T18:44:59.853476+0000","last_clean":"2021-01-15T18:44:59.853476+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:44:59.853476+0000","last_undegraded":"2021-01-15T18:44:59.853476+0000","last_fullsized":"2021-01-15T18:44:59.853476+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5581,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31785126,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5650","reported_seq":"5777","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:03.500496+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:03.500496+0000","last_peered":"2021-01-15T18:45:03.500496+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:03.500496+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5650,"ondisk_log_size":5650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31469675,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5649,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2449}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5766","reported_seq":"5938","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.833826+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:44:59.833826+0000","last_peered":"2021-01-15T18:44:59.833826+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:44:59.833826+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5766,"ondisk_log_size":5766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32980992,"num_objects":4026,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2655}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5776","reported_seq":"6368","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.831278+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:44:59.831278+0000","last_peered":"2021-01-15T18:44:59.831278+0000","last_clean":"2021-01-15T18:44:59.831278+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:44:59.831278+0000","last_undegraded":"2021-01-15T18:44:59.831278+0000","last_fullsized":"2021-01-15T18:44:59.831278+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5776,"ondisk_log_size":5776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33194150,"num_objects":4053,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5579","reported_seq":"5790","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.792696+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:44:59.792696+0000","last_peered":"2021-01-15T18:44:59.792696+0000","last_clean":"2021-01-15T18:44:59.792696+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:44:59.792696+0000","last_undegraded":"2021-01-15T18:44:59.792696+0000","last_fullsized":"2021-01-15T18:44:59.792696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5579,"ondisk_log_size":5579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32333990,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5665","reported_seq":"5843","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.770023+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:44:59.770023+0000","last_peered":"2021-01-15T18:44:59.770023+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:44:59.770023+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5665,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33259686,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5848","reported_seq":"5973","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:03.538153+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:03.538153+0000","last_peered":"2021-01-15T18:45:03.538153+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:03.538153+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5848,"ondisk_log_size":5848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32837715,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5847,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5704","reported_seq":"5886","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.792501+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:44:59.792501+0000","last_peered":"2021-01-15T18:44:59.792501+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:44:59.792501+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5704,"ondisk_log_size":5704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32968787,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5701","reported_seq":"8642","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:44:59.688178+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:44:59.688178+0000","last_peered":"2021-01-15T18:44:59.688178+0000","last_clean":"2021-01-15T18:44:59.688178+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:44:59.688178+0000","last_undegraded":"2021-01-15T18:44:59.688178+0000","last_fullsized":"2021-01-15T18:44:59.688178+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5701,"ondisk_log_size":5701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33231097,"num_objects":4058,"num_object_clones":0,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5698,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5597","reported_seq":"5769","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.850278+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:44:59.850278+0000","last_peered":"2021-01-15T18:44:59.850278+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:44:59.850278+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5597,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32415910,"num_objects":3958,"num_object_clones":0,"num_object_copies":15832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5647","reported_seq":"7284","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.385638+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:04.385638+0000","last_peered":"2021-01-15T18:45:04.385638+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:04.385638+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5647,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32112806,"num_objects":3921,"num_object_clones":0,"num_object_copies":15684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6356992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5613","reported_seq":"8508","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.255368+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:04.255368+0000","last_peered":"2021-01-15T18:45:04.255368+0000","last_clean":"2021-01-15T18:45:04.255368+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:04.255368+0000","last_undegraded":"2021-01-15T18:45:04.255368+0000","last_fullsized":"2021-01-15T18:45:04.255368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5613,"ondisk_log_size":5613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31637670,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5611,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5869","reported_seq":"5972","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.083295+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:04.083295+0000","last_peered":"2021-01-15T18:45:04.083295+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:04.083295+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5869,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32907430,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5867,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5894","reported_seq":"6080","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:44:59.631015+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:44:59.631015+0000","last_peered":"2021-01-15T18:44:59.631015+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:44:59.631015+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5894,"ondisk_log_size":5894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33980748,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5891,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5644","reported_seq":"8509","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:03.417640+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:03.417640+0000","last_peered":"2021-01-15T18:45:03.417640+0000","last_clean":"2021-01-15T18:45:03.417640+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:03.417640+0000","last_undegraded":"2021-01-15T18:45:03.417640+0000","last_fullsized":"2021-01-15T18:45:03.417640+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5644,"ondisk_log_size":5644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31908006,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5642,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":521398572,"num_objects":63662,"num_object_clones":0,"num_object_copies":254648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91063,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8451,"num_bytes_recovered":65282048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1017749504,"data_stored":1017749504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91090,"ondisk_log_size":91090,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435172,"kb_used_data":256388,"kb_used_omap":0,"kb_used_meta":178751,"kb_avail":93936668,"statfs":{"total":96636764160,"available":96191148032,"internally_reserved":0,"allocated":262541312,"data_stored":262327134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183041076},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.0269999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.125},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.212}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":1.833},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":3.3149999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429576,"kb_used_data":253112,"kb_used_omap":0,"kb_used_meta":176448,"kb_avail":93942264,"statfs":{"total":96636764160,"available":96196878336,"internally_reserved":0,"allocated":259186688,"data_stored":258972510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":180682752},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":27.870000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":15.721}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":15.609},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":16.904},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":15.856999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280583,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441296,"kb_used_data":252196,"kb_used_omap":0,"kb_used_meta":189055,"kb_avail":93930544,"statfs":{"total":96636764160,"available":96184877056,"internally_reserved":0,"allocated":258248704,"data_stored":258034526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":193592372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,4,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":6.1600000000000001},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.3609999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":7.0599999999999996},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.3970000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.6890000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":9.8659999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388732,"kb_used_data":233896,"kb_used_omap":0,"kb_used_meta":154815,"kb_avail":93983108,"statfs":{"total":96636764160,"available":96238702592,"internally_reserved":0,"allocated":239509504,"data_stored":239295326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":158530612},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.7579999999999991},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.843}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.755},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":7.9740000000000002}]}]}]}} 2021-01-15T18:45:06.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:07.194 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:07.194 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12914/251680 objects degraded (5.131%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":357,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62920,"data_bytes":515320108,"bytes_used":1746354176,"bytes_avail":384800702464,"bytes_total":386547056640,"degraded_objects":12914,"degraded_total":251680,"degraded_ratio":0.051311188811188811,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":119384,"recovering_keys_per_sec":0,"num_objects_recovered":316,"num_bytes_recovered":1433600,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":249},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2021-01-15T18:45:06.527228+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:45:07.219 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:07.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:08.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:08.184 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2021-01-15T18:45:06.527132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":515320108,"num_objects":62920,"num_object_clones":0,"num_object_copies":251680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91805,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8493,"num_bytes_recovered":65282048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91832,"ondisk_log_size":91832,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1705424,"kb_used_data":990280,"kb_used_omap":2,"kb_used_meta":715005,"kb_avail":375781936,"statfs":{"total":386547056640,"available":384800702464,"internally_reserved":0,"allocated":1014046720,"data_stored":1013190008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":732165276},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,13,23,14,0,0,5,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24543232,"num_objects":-2996,"num_object_clones":0,"num_object_copies":-11984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008301"},"pg_stats":[{"pgid":"1.f","version":"40'5627","reported_seq":"5833","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.879256+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:04.879256+0000","last_peered":"2021-01-15T18:45:04.879256+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:04.879256+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5627,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31793152,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5650","reported_seq":"8574","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.673948+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:04.673948+0000","last_peered":"2021-01-15T18:45:04.673948+0000","last_clean":"2021-01-15T18:45:04.673948+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:04.673948+0000","last_undegraded":"2021-01-15T18:45:04.673948+0000","last_fullsized":"2021-01-15T18:45:04.673948+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5650,"ondisk_log_size":5650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31219878,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5650","reported_seq":"5777","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:03.500496+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:03.500496+0000","last_peered":"2021-01-15T18:45:03.500496+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:03.500496+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5650,"ondisk_log_size":5650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31469675,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5649,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2449}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5849","reported_seq":"6035","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.788254+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:04.788254+0000","last_peered":"2021-01-15T18:45:04.788254+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:04.788254+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5849,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32301056,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5864","reported_seq":"6455","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.840969+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:04.840969+0000","last_peered":"2021-01-15T18:45:04.840969+0000","last_clean":"2021-01-15T18:45:04.840969+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:04.840969+0000","last_undegraded":"2021-01-15T18:45:04.840969+0000","last_fullsized":"2021-01-15T18:45:04.840969+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5864,"ondisk_log_size":5864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32473254,"num_objects":3965,"num_object_clones":0,"num_object_copies":15860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5862,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5654","reported_seq":"5865","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.850965+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:04.850965+0000","last_peered":"2021-01-15T18:45:04.850965+0000","last_clean":"2021-01-15T18:45:04.850965+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:04.850965+0000","last_undegraded":"2021-01-15T18:45:04.850965+0000","last_fullsized":"2021-01-15T18:45:04.850965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5654,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31719590,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5735","reported_seq":"5926","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.870259+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:04.870259+0000","last_peered":"2021-01-15T18:45:04.870259+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:04.870259+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5735,"ondisk_log_size":5735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32686246,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2666}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5848","reported_seq":"5973","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:03.538153+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:03.538153+0000","last_peered":"2021-01-15T18:45:03.538153+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:03.538153+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5848,"ondisk_log_size":5848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32837715,"num_objects":4009,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5847,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2581}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5786","reported_seq":"5984","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.885357+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:04.885357+0000","last_peered":"2021-01-15T18:45:04.885357+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:04.885357+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5786,"ondisk_log_size":5786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32297043,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5785,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5789","reported_seq":"8730","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.839422+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:04.839422+0000","last_peered":"2021-01-15T18:45:04.839422+0000","last_clean":"2021-01-15T18:45:04.839422+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:04.839422+0000","last_undegraded":"2021-01-15T18:45:04.839422+0000","last_fullsized":"2021-01-15T18:45:04.839422+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5789,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32510201,"num_objects":3970,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5650","reported_seq":"5836","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.575393+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:04.575393+0000","last_peered":"2021-01-15T18:45:04.575393+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:04.575393+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5650,"ondisk_log_size":5650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31981734,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5647","reported_seq":"7284","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.385638+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:04.385638+0000","last_peered":"2021-01-15T18:45:04.385638+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:04.385638+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5647,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32112806,"num_objects":3921,"num_object_clones":0,"num_object_copies":15684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":6356992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5613","reported_seq":"8508","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:04.255368+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:04.255368+0000","last_peered":"2021-01-15T18:45:04.255368+0000","last_clean":"2021-01-15T18:45:04.255368+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:04.255368+0000","last_undegraded":"2021-01-15T18:45:04.255368+0000","last_fullsized":"2021-01-15T18:45:04.255368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5613,"ondisk_log_size":5613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31637670,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5611,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5869","reported_seq":"5972","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.083295+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:04.083295+0000","last_peered":"2021-01-15T18:45:04.083295+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:04.083295+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5869,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32907430,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5867,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'5957","reported_seq":"6149","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:04.669918+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:04.669918+0000","last_peered":"2021-01-15T18:45:04.669918+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:04.669918+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5957,"ondisk_log_size":5957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33464652,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2665}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5644","reported_seq":"8509","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:03.417640+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:03.417640+0000","last_peered":"2021-01-15T18:45:03.417640+0000","last_clean":"2021-01-15T18:45:03.417640+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:03.417640+0000","last_undegraded":"2021-01-15T18:45:03.417640+0000","last_fullsized":"2021-01-15T18:45:03.417640+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5644,"ondisk_log_size":5644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31908006,"num_objects":3896,"num_object_clones":0,"num_object_copies":15584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5642,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":515320108,"num_objects":62920,"num_object_clones":0,"num_object_copies":251680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91805,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8493,"num_bytes_recovered":65282048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1011089408,"data_stored":1011089408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91832,"ondisk_log_size":91832,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215171,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437288,"kb_used_data":252468,"kb_used_omap":0,"kb_used_meta":184767,"kb_avail":93934552,"statfs":{"total":96636764160,"available":96188981248,"internally_reserved":0,"allocated":258527232,"data_stored":258313054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":189201460},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,4,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":3.3050000000000002,"5min":3.379,"15min":3.1320000000000001},"min":{"1min":0.56000000000000005,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.981,"5min":16.853999999999999,"15min":16.853999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.3500000000000001,"15min":3.4279999999999999},"min":{"1min":0.49199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.084,"5min":16.629000000000001,"15min":16.629000000000001},"last":1.0269999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.125},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.212}]},{"osd":2,"last update":"Fri Jan 15 18:44:00 2021","interfaces":[{"interface":"back","average":{"1min":4.2569999999999997,"5min":3.0600000000000001,"15min":2.3500000000000001},"min":{"1min":0.78900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.7669999999999995,"5min":15.164,"15min":15.164},"last":1.833},{"interface":"front","average":{"1min":6.3609999999999998,"5min":3.7370000000000001,"15min":3.1429999999999998},"min":{"1min":0.91500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":14.436,"5min":18.300000000000001,"15min":18.300000000000001},"last":3.3149999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429576,"kb_used_data":253112,"kb_used_omap":0,"kb_used_meta":176448,"kb_avail":93942264,"statfs":{"total":96636764160,"available":96196878336,"internally_reserved":0,"allocated":259186688,"data_stored":258972510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":180682752},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":27.870000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":15.721}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":15.609},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":16.904},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":15.856999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280583,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":441296,"kb_used_data":252196,"kb_used_omap":0,"kb_used_meta":189055,"kb_avail":93930544,"statfs":{"total":96636764160,"available":96184877056,"internally_reserved":0,"allocated":258248704,"data_stored":258034526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":193592372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,4,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":6.1600000000000001},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.3609999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":7.0599999999999996},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.3970000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":4.6890000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":9.8659999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397264,"kb_used_data":232504,"kb_used_omap":0,"kb_used_meta":164735,"kb_avail":93974576,"statfs":{"total":96636764160,"available":96229965824,"internally_reserved":0,"allocated":238084096,"data_stored":237869918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":168688692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.7579999999999991},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.843}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.755},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":7.9740000000000002}]}]}]}} 2021-01-15T18:45:11.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:12.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:12.277 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:12.300 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2021-01-15T18:45:10.531737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506710316,"num_objects":61869,"num_object_clones":0,"num_object_copies":247476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92856,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8604,"num_bytes_recovered":66076672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92883,"ondisk_log_size":92883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1687352,"kb_used_data":975732,"kb_used_omap":2,"kb_used_meta":711485,"kb_avail":375800008,"statfs":{"total":386547056640,"available":384819208192,"internally_reserved":0,"allocated":999149568,"data_stored":998292856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":728560796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,17,20,14,14,7,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27631616,"num_objects":-3373,"num_object_clones":0,"num_object_copies":-13492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3373,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009927"},"pg_stats":[{"pgid":"1.f","version":"40'5692","reported_seq":"5898","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.896579+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:09.896579+0000","last_peered":"2021-01-15T18:45:09.896579+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:09.896579+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5692,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5708","reported_seq":"8632","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.738742+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:09.738742+0000","last_peered":"2021-01-15T18:45:09.738742+0000","last_clean":"2021-01-15T18:45:09.738742+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:09.738742+0000","last_undegraded":"2021-01-15T18:45:09.738742+0000","last_fullsized":"2021-01-15T18:45:09.738742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5708,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30744742,"num_objects":3754,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5722","reported_seq":"5852","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:08.386078+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:08.386078+0000","last_peered":"2021-01-15T18:45:08.386078+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:08.386078+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5722,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30879851,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5721,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5915","reported_seq":"6100","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.853113+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:09.853113+0000","last_peered":"2021-01-15T18:45:09.853113+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:09.853113+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5915,"ondisk_log_size":5915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31760384,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5947","reported_seq":"6539","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.918506+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:09.918506+0000","last_peered":"2021-01-15T18:45:09.918506+0000","last_clean":"2021-01-15T18:45:09.918506+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:09.918506+0000","last_undegraded":"2021-01-15T18:45:09.918506+0000","last_fullsized":"2021-01-15T18:45:09.918506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5947,"ondisk_log_size":5947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31793318,"num_objects":3882,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5718","reported_seq":"5929","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.918555+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:09.918555+0000","last_peered":"2021-01-15T18:45:09.918555+0000","last_clean":"2021-01-15T18:45:09.918555+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:09.918555+0000","last_undegraded":"2021-01-15T18:45:09.918555+0000","last_fullsized":"2021-01-15T18:45:09.918555+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5718,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31195302,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5804","reported_seq":"6000","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.785745+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:09.785745+0000","last_peered":"2021-01-15T18:45:09.785745+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:09.785745+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5804,"ondisk_log_size":5804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32120998,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5802,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2599}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5917","reported_seq":"6047","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:08.426158+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:08.426158+0000","last_peered":"2021-01-15T18:45:08.426158+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:08.426158+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5917,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32272467,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5854","reported_seq":"6054","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.786200+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:09.786200+0000","last_peered":"2021-01-15T18:45:09.786200+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:09.786200+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5854,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31739987,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5851","reported_seq":"8792","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.858226+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:09.858226+0000","last_peered":"2021-01-15T18:45:09.858226+0000","last_clean":"2021-01-15T18:45:09.858226+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:09.858226+0000","last_undegraded":"2021-01-15T18:45:09.858226+0000","last_fullsized":"2021-01-15T18:45:09.858226+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5851,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32002297,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5712","reported_seq":"5904","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.677463+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:09.677463+0000","last_peered":"2021-01-15T18:45:09.677463+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:09.677463+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5712,"ondisk_log_size":5712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31473830,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5710,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5709","reported_seq":"7540","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.387297+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:09.387297+0000","last_peered":"2021-01-15T18:45:09.387297+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:09.387297+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5709,"ondisk_log_size":5709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31604902,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5707,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7151616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5669","reported_seq":"8564","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.264928+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:09.264928+0000","last_peered":"2021-01-15T18:45:09.264928+0000","last_clean":"2021-01-15T18:45:09.264928+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:09.264928+0000","last_undegraded":"2021-01-15T18:45:09.264928+0000","last_fullsized":"2021-01-15T18:45:09.264928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5669,"ondisk_log_size":5669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31178918,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5667,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5930","reported_seq":"6039","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.166752+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:09.166752+0000","last_peered":"2021-01-15T18:45:09.166752+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:09.166752+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5930,"ondisk_log_size":5930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32407718,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6025","reported_seq":"6221","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.868096+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:09.868096+0000","last_peered":"2021-01-15T18:45:09.868096+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:09.868096+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6025,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32907596,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5710","reported_seq":"8575","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:08.585464+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:08.585464+0000","last_peered":"2021-01-15T18:45:08.585464+0000","last_clean":"2021-01-15T18:45:08.585464+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:08.585464+0000","last_undegraded":"2021-01-15T18:45:08.585464+0000","last_fullsized":"2021-01-15T18:45:08.585464+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5710,"ondisk_log_size":5710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31367334,"num_objects":3830,"num_object_clones":0,"num_object_copies":15320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":506710316,"num_objects":61869,"num_object_clones":0,"num_object_copies":247476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92856,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8604,"num_bytes_recovered":66076672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":996712448,"data_stored":996712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92883,"ondisk_log_size":92883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426880,"kb_used_data":248092,"kb_used_omap":0,"kb_used_meta":178751,"kb_avail":93944960,"statfs":{"total":96636764160,"available":96199639040,"internally_reserved":0,"allocated":254046208,"data_stored":253832030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183041076},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,10,2,4,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":9.8629999999999995},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":10.125}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":1.4690000000000001},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.89100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435556,"kb_used_data":248772,"kb_used_omap":0,"kb_used_meta":186752,"kb_avail":93936284,"statfs":{"total":96636764160,"available":96190754816,"internally_reserved":0,"allocated":254742528,"data_stored":254528350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":191234048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":3.0350000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":3.7480000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.82999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.61899999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430968,"kb_used_data":247956,"kb_used_omap":0,"kb_used_meta":182975,"kb_avail":93940872,"statfs":{"total":96636764160,"available":96195452928,"internally_reserved":0,"allocated":253906944,"data_stored":253692766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":187366452},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,3,5,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":5.3280000000000003},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.7389999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.159}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.698},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":1.5329999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393948,"kb_used_data":230912,"kb_used_omap":0,"kb_used_meta":163007,"kb_avail":93977892,"statfs":{"total":96636764160,"available":96233361408,"internally_reserved":0,"allocated":236453888,"data_stored":236239710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166919220},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.7579999999999991},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.748}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.8480000000000001}]}]}]}} 2021-01-15T18:45:12.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:13.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:13.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12803/247476 objects degraded (5.173%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61869,"data_bytes":506710316,"bytes_used":1727848448,"bytes_avail":384819208192,"bytes_total":386547056640,"degraded_objects":12803,"degraded_total":247476,"degraded_ratio":0.051734309589616771,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":124142,"recovering_keys_per_sec":0,"num_objects_recovered":238,"num_bytes_recovered":1490944,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2021-01-15T18:45:06.527228+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:45:13.458 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:13.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:14.148 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:14.156 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:14.205 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2021-01-15T18:45:12.532268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506710316,"num_objects":61869,"num_object_clones":0,"num_object_copies":247476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92856,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8604,"num_bytes_recovered":66076672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92883,"ondisk_log_size":92883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1687352,"kb_used_data":975732,"kb_used_omap":2,"kb_used_meta":711485,"kb_avail":375800008,"statfs":{"total":386547056640,"available":384819208192,"internally_reserved":0,"allocated":999149568,"data_stored":998292856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":728560796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,17,20,14,14,7,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-2226,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009985"},"pg_stats":[{"pgid":"1.f","version":"40'5692","reported_seq":"5898","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.896579+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:09.896579+0000","last_peered":"2021-01-15T18:45:09.896579+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:09.896579+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5692,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5708","reported_seq":"8632","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.738742+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:09.738742+0000","last_peered":"2021-01-15T18:45:09.738742+0000","last_clean":"2021-01-15T18:45:09.738742+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:09.738742+0000","last_undegraded":"2021-01-15T18:45:09.738742+0000","last_fullsized":"2021-01-15T18:45:09.738742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5708,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30744742,"num_objects":3754,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5722","reported_seq":"5852","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:08.386078+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:08.386078+0000","last_peered":"2021-01-15T18:45:08.386078+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:08.386078+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5722,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30879851,"num_objects":3771,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5721,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5915","reported_seq":"6100","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.853113+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:09.853113+0000","last_peered":"2021-01-15T18:45:09.853113+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:09.853113+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5915,"ondisk_log_size":5915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31760384,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'5947","reported_seq":"6539","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.918506+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:09.918506+0000","last_peered":"2021-01-15T18:45:09.918506+0000","last_clean":"2021-01-15T18:45:09.918506+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:09.918506+0000","last_undegraded":"2021-01-15T18:45:09.918506+0000","last_fullsized":"2021-01-15T18:45:09.918506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5947,"ondisk_log_size":5947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31793318,"num_objects":3882,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5945,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5718","reported_seq":"5929","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.918555+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:09.918555+0000","last_peered":"2021-01-15T18:45:09.918555+0000","last_clean":"2021-01-15T18:45:09.918555+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:09.918555+0000","last_undegraded":"2021-01-15T18:45:09.918555+0000","last_fullsized":"2021-01-15T18:45:09.918555+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5718,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31195302,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5804","reported_seq":"6000","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.785745+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:09.785745+0000","last_peered":"2021-01-15T18:45:09.785745+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:09.785745+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5804,"ondisk_log_size":5804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32120998,"num_objects":3922,"num_object_clones":0,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5802,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2599}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5917","reported_seq":"6047","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:08.426158+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:08.426158+0000","last_peered":"2021-01-15T18:45:08.426158+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:08.426158+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5917,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32272467,"num_objects":3940,"num_object_clones":0,"num_object_copies":15760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5916,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5854","reported_seq":"6054","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.786200+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:09.786200+0000","last_peered":"2021-01-15T18:45:09.786200+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:09.786200+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5854,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31739987,"num_objects":3875,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5851","reported_seq":"8792","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.858226+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:09.858226+0000","last_peered":"2021-01-15T18:45:09.858226+0000","last_clean":"2021-01-15T18:45:09.858226+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:09.858226+0000","last_undegraded":"2021-01-15T18:45:09.858226+0000","last_fullsized":"2021-01-15T18:45:09.858226+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5851,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32002297,"num_objects":3908,"num_object_clones":0,"num_object_copies":15632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5712","reported_seq":"5904","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.677463+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:09.677463+0000","last_peered":"2021-01-15T18:45:09.677463+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:09.677463+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5712,"ondisk_log_size":5712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31473830,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5710,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5709","reported_seq":"7540","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.387297+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:09.387297+0000","last_peered":"2021-01-15T18:45:09.387297+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:09.387297+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5709,"ondisk_log_size":5709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31604902,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5707,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7151616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5669","reported_seq":"8564","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:09.264928+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:09.264928+0000","last_peered":"2021-01-15T18:45:09.264928+0000","last_clean":"2021-01-15T18:45:09.264928+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:09.264928+0000","last_undegraded":"2021-01-15T18:45:09.264928+0000","last_fullsized":"2021-01-15T18:45:09.264928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5669,"ondisk_log_size":5669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31178918,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5667,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'5930","reported_seq":"6039","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.166752+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:09.166752+0000","last_peered":"2021-01-15T18:45:09.166752+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:09.166752+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5930,"ondisk_log_size":5930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32407718,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6025","reported_seq":"6221","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:09.868096+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:09.868096+0000","last_peered":"2021-01-15T18:45:09.868096+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:09.868096+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6025,"ondisk_log_size":6025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32907596,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5710","reported_seq":"8575","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:08.585464+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:08.585464+0000","last_peered":"2021-01-15T18:45:08.585464+0000","last_clean":"2021-01-15T18:45:08.585464+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:08.585464+0000","last_undegraded":"2021-01-15T18:45:08.585464+0000","last_fullsized":"2021-01-15T18:45:08.585464+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5710,"ondisk_log_size":5710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31367334,"num_objects":3830,"num_object_clones":0,"num_object_copies":15320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":506710316,"num_objects":61869,"num_object_clones":0,"num_object_copies":247476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92856,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8604,"num_bytes_recovered":66076672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":996712448,"data_stored":996712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92883,"ondisk_log_size":92883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215172,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426880,"kb_used_data":248092,"kb_used_omap":0,"kb_used_meta":178751,"kb_avail":93944960,"statfs":{"total":96636764160,"available":96199639040,"internally_reserved":0,"allocated":254046208,"data_stored":253832030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183041076},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,10,2,4,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":9.8629999999999995},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":10.125}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":1.4690000000000001},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.89100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435556,"kb_used_data":248772,"kb_used_omap":0,"kb_used_meta":186752,"kb_avail":93936284,"statfs":{"total":96636764160,"available":96190754816,"internally_reserved":0,"allocated":254742528,"data_stored":254528350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":191234048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":3.0350000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":3.7480000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.82999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.61899999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430968,"kb_used_data":247956,"kb_used_omap":0,"kb_used_meta":182975,"kb_avail":93940872,"statfs":{"total":96636764160,"available":96195452928,"internally_reserved":0,"allocated":253906944,"data_stored":253692766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":187366452},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,3,5,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":5.3280000000000003},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.7389999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.159}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":2.698},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":1.5329999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393948,"kb_used_data":230912,"kb_used_omap":0,"kb_used_meta":163007,"kb_avail":93977892,"statfs":{"total":96636764160,"available":96233361408,"internally_reserved":0,"allocated":236453888,"data_stored":236239710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166919220},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":8.7579999999999991},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.748}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.8480000000000001}]}]}]}} 2021-01-15T18:45:17.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:19.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:19.254 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:19.289 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2021-01-15T18:45:18.533724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":498420012,"num_objects":60857,"num_object_clones":0,"num_object_copies":243428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93868,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8761,"num_bytes_recovered":67059712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93895,"ondisk_log_size":93895,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1702488,"kb_used_data":962508,"kb_used_omap":2,"kb_used_meta":739901,"kb_avail":375784872,"statfs":{"total":386547056640,"available":384803708928,"internally_reserved":0,"allocated":985608192,"data_stored":984751480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":757658780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,14,16,2,0,5,0,1,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":97,"apply_latency_ms":97,"commit_latency_ns":97000000,"apply_latency_ns":97000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16900096,"num_objects":-2063,"num_object_clones":0,"num_object_copies":-8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2063,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1777664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006591"},"pg_stats":[{"pgid":"1.f","version":"40'5747","reported_seq":"5960","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.800948+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:14.800948+0000","last_peered":"2021-01-15T18:45:14.800948+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:14.800948+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5747,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30810112,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5768","reported_seq":"8692","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:14.905171+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:14.905171+0000","last_peered":"2021-01-15T18:45:14.905171+0000","last_clean":"2021-01-15T18:45:14.905171+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:14.905171+0000","last_undegraded":"2021-01-15T18:45:14.905171+0000","last_fullsized":"2021-01-15T18:45:14.905171+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5768,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30253222,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5766,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5779","reported_seq":"5908","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:13.491429+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:13.491429+0000","last_peered":"2021-01-15T18:45:13.491429+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:13.491429+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5779,"ondisk_log_size":5779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30412907,"num_objects":3714,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5778,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'5965","reported_seq":"6161","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.713629+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:14.713629+0000","last_peered":"2021-01-15T18:45:14.713629+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:14.713629+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5965,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31350784,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2468}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6007","reported_seq":"6599","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:14.748440+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:14.748440+0000","last_peered":"2021-01-15T18:45:14.748440+0000","last_clean":"2021-01-15T18:45:14.748440+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:14.748440+0000","last_undegraded":"2021-01-15T18:45:14.748440+0000","last_fullsized":"2021-01-15T18:45:14.748440+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6007,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31301798,"num_objects":3822,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5775","reported_seq":"5986","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:14.844477+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:14.844477+0000","last_peered":"2021-01-15T18:45:14.844477+0000","last_clean":"2021-01-15T18:45:14.844477+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:14.844477+0000","last_undegraded":"2021-01-15T18:45:14.844477+0000","last_fullsized":"2021-01-15T18:45:14.844477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5775,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30728358,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5878","reported_seq":"6080","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.792285+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:14.792285+0000","last_peered":"2021-01-15T18:45:14.792285+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:14.792285+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5878,"ondisk_log_size":5878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31514790,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2528}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'5994","reported_seq":"6130","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:13.437266+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:13.437266+0000","last_peered":"2021-01-15T18:45:13.437266+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:13.437266+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5994,"ondisk_log_size":5994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31641683,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5993,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5925","reported_seq":"6133","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.799566+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:14.799566+0000","last_peered":"2021-01-15T18:45:14.799566+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:14.799566+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5925,"ondisk_log_size":5925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31158355,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5930","reported_seq":"8869","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:14.878262+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:14.878262+0000","last_peered":"2021-01-15T18:45:14.878262+0000","last_clean":"2021-01-15T18:45:14.878262+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:14.878262+0000","last_undegraded":"2021-01-15T18:45:14.878262+0000","last_fullsized":"2021-01-15T18:45:14.878262+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5930,"ondisk_log_size":5930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31355129,"num_objects":3829,"num_object_clones":0,"num_object_copies":15316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5774","reported_seq":"5982","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.858567+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:14.858567+0000","last_peered":"2021-01-15T18:45:14.858567+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:14.858567+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5774,"ondisk_log_size":5774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30965926,"num_objects":3781,"num_object_clones":0,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5772,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5767","reported_seq":"7838","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.280459+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:14.280459+0000","last_peered":"2021-01-15T18:45:14.280459+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:14.280459+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5767,"ondisk_log_size":5767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31129766,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5765,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":8134656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5724","reported_seq":"8619","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:14.173263+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:14.173263+0000","last_peered":"2021-01-15T18:45:14.173263+0000","last_clean":"2021-01-15T18:45:14.173263+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:14.173263+0000","last_undegraded":"2021-01-15T18:45:14.173263+0000","last_fullsized":"2021-01-15T18:45:14.173263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5724,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30728358,"num_objects":3752,"num_object_clones":0,"num_object_copies":15008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6008","reported_seq":"6125","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.070678+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:14.070678+0000","last_peered":"2021-01-15T18:45:14.070678+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:14.070678+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6008,"ondisk_log_size":6008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31768742,"num_objects":3879,"num_object_clones":0,"num_object_copies":15516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6006,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6093","reported_seq":"6301","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:14.722188+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:14.722188+0000","last_peered":"2021-01-15T18:45:14.722188+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:14.722188+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6093,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32350540,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6090,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2537}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5761","reported_seq":"8626","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:13.252246+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:13.252246+0000","last_peered":"2021-01-15T18:45:13.252246+0000","last_clean":"2021-01-15T18:45:13.252246+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:13.252246+0000","last_undegraded":"2021-01-15T18:45:13.252246+0000","last_fullsized":"2021-01-15T18:45:13.252246+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5761,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30949542,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":498420012,"num_objects":60857,"num_object_clones":0,"num_object_copies":243428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12646,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93868,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8761,"num_bytes_recovered":67059712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983044096,"data_stored":983044096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93895,"ondisk_log_size":93895,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215173,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433124,"kb_used_data":244112,"kb_used_omap":0,"kb_used_meta":188991,"kb_avail":93938716,"statfs":{"total":96636764160,"available":96193245184,"internally_reserved":0,"allocated":249970688,"data_stored":249756510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":193526836},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,2,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":9.8629999999999995},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":10.125}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.788}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":6.4420000000000002},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.234}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429472,"kb_used_data":244608,"kb_used_omap":0,"kb_used_meta":184832,"kb_avail":93942368,"statfs":{"total":96636764160,"available":96196984832,"internally_reserved":0,"allocated":250478592,"data_stored":250264414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":189267968},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":10.545999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.94499999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437196,"kb_used_data":243964,"kb_used_omap":0,"kb_used_meta":193215,"kb_avail":93934644,"statfs":{"total":96636764160,"available":96189075456,"internally_reserved":0,"allocated":249819136,"data_stored":249604958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":197852212},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,5,7,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":0.58099999999999996},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":1.9590000000000001},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":2.0630000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":8.6099999999999994},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.71099999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402696,"kb_used_data":229824,"kb_used_omap":0,"kb_used_meta":172863,"kb_avail":93969144,"statfs":{"total":96636764160,"available":96224403456,"internally_reserved":0,"allocated":235339776,"data_stored":235125598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":177011764},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.0880000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":7.5229999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.145},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":8.6549999999999994}]}]}]}} 2021-01-15T18:45:19.291 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:20.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:20.328 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12646/243428 objects degraded (5.195%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":370,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":60857,"data_bytes":498420012,"bytes_used":1743347712,"bytes_avail":384803708928,"bytes_total":386547056640,"degraded_objects":12646,"degraded_total":243428,"degraded_ratio":0.051949652463972915,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":148057,"recovering_keys_per_sec":0,"num_objects_recovered":268,"num_bytes_recovered":1777664,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2021-01-15T18:45:06.527228+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:45:20.361 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:20.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:21.108 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:21.144 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:21.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2021-01-15T18:45:20.538073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489761068,"num_objects":59800,"num_object_clones":0,"num_object_copies":239200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94925,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8890,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94952,"ondisk_log_size":94952,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1664140,"kb_used_data":948400,"kb_used_omap":2,"kb_used_meta":715581,"kb_avail":375823220,"statfs":{"total":386547056640,"available":384842977280,"internally_reserved":0,"allocated":971161600,"data_stored":970304888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":732755100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,12,11,13,18,1,0,3,2,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25559040,"num_objects":-3120,"num_object_clones":0,"num_object_copies":-12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010410"},"pg_stats":[{"pgid":"1.f","version":"40'5819","reported_seq":"6043","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.834748+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:19.834748+0000","last_peered":"2021-01-15T18:45:19.834748+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:19.834748+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5819,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30220288,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5832","reported_seq":"8756","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.904449+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:19.904449+0000","last_peered":"2021-01-15T18:45:19.904449+0000","last_clean":"2021-01-15T18:45:19.904449+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:19.904449+0000","last_undegraded":"2021-01-15T18:45:19.904449+0000","last_fullsized":"2021-01-15T18:45:19.904449+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5832,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29728934,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5849","reported_seq":"5989","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:18.457830+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:18.457830+0000","last_peered":"2021-01-15T18:45:18.457830+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:18.457830+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5849,"ondisk_log_size":5849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29839467,"num_objects":3644,"num_object_clones":0,"num_object_copies":14576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5848,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'6031","reported_seq":"6231","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.918058+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:19.918058+0000","last_peered":"2021-01-15T18:45:19.918058+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:19.918058+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6031,"ondisk_log_size":6031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30810112,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6086","reported_seq":"6678","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.874881+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:19.874881+0000","last_peered":"2021-01-15T18:45:19.874881+0000","last_clean":"2021-01-15T18:45:19.874881+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:19.874881+0000","last_undegraded":"2021-01-15T18:45:19.874881+0000","last_fullsized":"2021-01-15T18:45:19.874881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6086,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30654630,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5846","reported_seq":"6057","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.706028+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:19.706028+0000","last_peered":"2021-01-15T18:45:19.706028+0000","last_clean":"2021-01-15T18:45:19.706028+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:19.706028+0000","last_undegraded":"2021-01-15T18:45:19.706028+0000","last_fullsized":"2021-01-15T18:45:19.706028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5846,"ondisk_log_size":5846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30146726,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5935","reported_seq":"6143","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.653845+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:19.653845+0000","last_peered":"2021-01-15T18:45:19.653845+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:19.653845+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5935,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31047846,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6050","reported_seq":"6194","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:18.445369+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:18.445369+0000","last_peered":"2021-01-15T18:45:18.445369+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:18.445369+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6050,"ondisk_log_size":6050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31182931,"num_objects":3807,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5991","reported_seq":"6209","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.786309+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:19.786309+0000","last_peered":"2021-01-15T18:45:19.786309+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:19.786309+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5991,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30617683,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5993","reported_seq":"8933","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.876388+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:19.876388+0000","last_peered":"2021-01-15T18:45:19.876388+0000","last_clean":"2021-01-15T18:45:19.876388+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:19.876388+0000","last_undegraded":"2021-01-15T18:45:19.876388+0000","last_fullsized":"2021-01-15T18:45:19.876388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5993,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30839033,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5844","reported_seq":"6056","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.861579+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:19.861579+0000","last_peered":"2021-01-15T18:45:19.861579+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:19.861579+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5844,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30392486,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5838","reported_seq":"8112","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.339414+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:19.339414+0000","last_peered":"2021-01-15T18:45:19.339414+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:19.339414+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5838,"ondisk_log_size":5838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30548134,"num_objects":3730,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5836,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1105,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5781","reported_seq":"8676","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.280790+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:19.280790+0000","last_peered":"2021-01-15T18:45:19.280790+0000","last_clean":"2021-01-15T18:45:19.280790+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:19.280790+0000","last_undegraded":"2021-01-15T18:45:19.280790+0000","last_fullsized":"2021-01-15T18:45:19.280790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5781,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30261414,"num_objects":3695,"num_object_clones":0,"num_object_copies":14780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6076","reported_seq":"6193","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.181661+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:19.181661+0000","last_peered":"2021-01-15T18:45:19.181661+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:19.181661+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6076,"ondisk_log_size":6076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31211686,"num_objects":3811,"num_object_clones":0,"num_object_copies":15244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6074,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6157","reported_seq":"6367","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.741167+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:19.741167+0000","last_peered":"2021-01-15T18:45:19.741167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:19.741167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6157,"ondisk_log_size":6157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31826252,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5824","reported_seq":"8689","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:18.291107+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:18.291107+0000","last_peered":"2021-01-15T18:45:18.291107+0000","last_clean":"2021-01-15T18:45:18.291107+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:18.291107+0000","last_undegraded":"2021-01-15T18:45:18.291107+0000","last_fullsized":"2021-01-15T18:45:18.291107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5824,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30433446,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":489761068,"num_objects":59800,"num_object_clones":0,"num_object_copies":239200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":94925,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8890,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":968470528,"data_stored":968470528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94952,"ondisk_log_size":94952,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422904,"kb_used_data":239868,"kb_used_omap":0,"kb_used_meta":182975,"kb_avail":93948936,"statfs":{"total":96636764160,"available":96203710464,"internally_reserved":0,"allocated":245624832,"data_stored":245410654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":187366452},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,4,6,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":5.6859999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":5.774}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.70899999999999996},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.5759999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419212,"kb_used_data":240492,"kb_used_omap":0,"kb_used_meta":178688,"kb_avail":93952628,"statfs":{"total":96636764160,"available":96207491072,"internally_reserved":0,"allocated":246263808,"data_stored":246049630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":182976512},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,7,1,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.84199999999999997},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.82899999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426972,"kb_used_data":239744,"kb_used_omap":0,"kb_used_meta":187199,"kb_avail":93944868,"statfs":{"total":96636764160,"available":96199544832,"internally_reserved":0,"allocated":245497856,"data_stored":245283678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":191691828},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,2,4,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":9.0890000000000004},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":9.173}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":4.3490000000000002},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":8.8239999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":8.7159999999999993},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":8.6170000000000009}]}]},{"osd":1,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395052,"kb_used_data":228296,"kb_used_omap":0,"kb_used_meta":166719,"kb_avail":93976788,"statfs":{"total":96636764160,"available":96232230912,"internally_reserved":0,"allocated":233775104,"data_stored":233560926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170720308},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":1.675},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.1839999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.27},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.7749999999999999}]}]}]}} 2021-01-15T18:45:24.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:25.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:25.453 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:25.497 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2021-01-15T18:45:24.541073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486451500,"num_objects":59396,"num_object_clones":0,"num_object_copies":237584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95329,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8976,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95356,"ondisk_log_size":95356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1659936,"kb_used_data":939804,"kb_used_omap":2,"kb_used_meta":719933,"kb_avail":375827424,"statfs":{"total":386547056640,"available":384847282176,"internally_reserved":0,"allocated":962359296,"data_stored":961502584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":737211548},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,8,10,21,9,0,0,0,2,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":94,"apply_latency_ms":94,"commit_latency_ns":94000000,"apply_latency_ns":94000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20258816,"num_objects":-2473,"num_object_clones":0,"num_object_copies":-9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2473,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008804"},"pg_stats":[{"pgid":"1.f","version":"40'5819","reported_seq":"6043","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.834748+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:19.834748+0000","last_peered":"2021-01-15T18:45:19.834748+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:19.834748+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5819,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30220288,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5832","reported_seq":"8756","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.904449+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:19.904449+0000","last_peered":"2021-01-15T18:45:19.904449+0000","last_clean":"2021-01-15T18:45:19.904449+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:19.904449+0000","last_undegraded":"2021-01-15T18:45:19.904449+0000","last_fullsized":"2021-01-15T18:45:19.904449+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5832,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29728934,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5920","reported_seq":"6062","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:23.468866+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:23.468866+0000","last_peered":"2021-01-15T18:45:23.468866+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:23.468866+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5920,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29257835,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5919,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2186}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'6031","reported_seq":"6231","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.918058+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:19.918058+0000","last_peered":"2021-01-15T18:45:19.918058+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:19.918058+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6031,"ondisk_log_size":6031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30810112,"num_objects":3761,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6086","reported_seq":"6678","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.874881+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:19.874881+0000","last_peered":"2021-01-15T18:45:19.874881+0000","last_clean":"2021-01-15T18:45:19.874881+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:19.874881+0000","last_undegraded":"2021-01-15T18:45:19.874881+0000","last_fullsized":"2021-01-15T18:45:19.874881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6086,"ondisk_log_size":6086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30654630,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5846","reported_seq":"6057","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.706028+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:19.706028+0000","last_peered":"2021-01-15T18:45:19.706028+0000","last_clean":"2021-01-15T18:45:19.706028+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:19.706028+0000","last_undegraded":"2021-01-15T18:45:19.706028+0000","last_fullsized":"2021-01-15T18:45:19.706028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5846,"ondisk_log_size":5846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30146726,"num_objects":3681,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'5935","reported_seq":"6143","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.653845+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:19.653845+0000","last_peered":"2021-01-15T18:45:19.653845+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:19.653845+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5935,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31047846,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6113","reported_seq":"6259","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:23.291924+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:23.291924+0000","last_peered":"2021-01-15T18:45:23.291924+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:23.291924+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6113,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30666835,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6112,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'5991","reported_seq":"6209","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.786309+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:19.786309+0000","last_peered":"2021-01-15T18:45:19.786309+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:19.786309+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5991,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30617683,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2388}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'5993","reported_seq":"8933","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:19.876388+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:19.876388+0000","last_peered":"2021-01-15T18:45:19.876388+0000","last_clean":"2021-01-15T18:45:19.876388+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:19.876388+0000","last_undegraded":"2021-01-15T18:45:19.876388+0000","last_fullsized":"2021-01-15T18:45:19.876388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5993,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30839033,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5844","reported_seq":"6056","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.861579+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:19.861579+0000","last_peered":"2021-01-15T18:45:19.861579+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:19.861579+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5844,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30392486,"num_objects":3711,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2405}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5916","reported_seq":"8357","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.348750+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:24.348750+0000","last_peered":"2021-01-15T18:45:24.348750+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:24.348750+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5916,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29909158,"num_objects":3652,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5849","reported_seq":"8744","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.301862+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:24.301862+0000","last_peered":"2021-01-15T18:45:24.301862+0000","last_clean":"2021-01-15T18:45:24.301862+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:24.301862+0000","last_undegraded":"2021-01-15T18:45:24.301862+0000","last_fullsized":"2021-01-15T18:45:24.301862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5849,"ondisk_log_size":5849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29704358,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5847,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6143","reported_seq":"6261","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.126613+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:24.126613+0000","last_peered":"2021-01-15T18:45:24.126613+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:24.126613+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6143,"ondisk_log_size":6143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30662822,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6157","reported_seq":"6367","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:19.741167+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:19.741167+0000","last_peered":"2021-01-15T18:45:19.741167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:19.741167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6157,"ondisk_log_size":6157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31826252,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5881","reported_seq":"8745","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:23.409114+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:23.409114+0000","last_peered":"2021-01-15T18:45:23.409114+0000","last_clean":"2021-01-15T18:45:23.409114+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:23.409114+0000","last_undegraded":"2021-01-15T18:45:23.409114+0000","last_fullsized":"2021-01-15T18:45:23.409114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5881,"ondisk_log_size":5881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29966502,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":486451500,"num_objects":59396,"num_object_clones":0,"num_object_copies":237584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95329,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8976,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":959737856,"data_stored":959737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95356,"ondisk_log_size":95356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215174,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422904,"kb_used_data":239868,"kb_used_omap":0,"kb_used_meta":182975,"kb_avail":93948936,"statfs":{"total":96636764160,"available":96203710464,"internally_reserved":0,"allocated":245624832,"data_stored":245410654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":187366452},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,4,6,4,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":5.6859999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":5.774}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.70899999999999996},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":5.5759999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425296,"kb_used_data":236216,"kb_used_omap":0,"kb_used_meta":189056,"kb_avail":93946544,"statfs":{"total":96636764160,"available":96201261056,"internally_reserved":0,"allocated":241885184,"data_stored":241671006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":193593344},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,6,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":3.4409999999999998},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":2.8570000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":1.222},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":3.2869999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416684,"kb_used_data":235424,"kb_used_omap":0,"kb_used_meta":181183,"kb_avail":93955156,"statfs":{"total":96636764160,"available":96210079744,"internally_reserved":0,"allocated":241074176,"data_stored":240859998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":185531444},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,9,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":7.1349999999999998},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":6.8959999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.512},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":2.4020000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395052,"kb_used_data":228296,"kb_used_omap":0,"kb_used_meta":166719,"kb_avail":93976788,"statfs":{"total":96636764160,"available":96232230912,"internally_reserved":0,"allocated":233775104,"data_stored":233560926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170720308},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":1.675},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.1839999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.27},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.7749999999999999}]}]}]}} 2021-01-15T18:45:25.499 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:26.858 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:26.859 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12409/234952 objects degraded (5.282%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":377,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58738,"data_bytes":481061164,"bytes_used":1722851328,"bytes_avail":384824205312,"bytes_total":386547056640,"degraded_objects":12409,"degraded_total":234952,"degraded_ratio":0.052815043072627604,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":48433,"recovering_keys_per_sec":0,"num_objects_recovered":267,"num_bytes_recovered":581632,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":229},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:26.872 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:26.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:27.625 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:27.636 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:27.686 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2021-01-15T18:45:26.541575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481061164,"num_objects":58738,"num_object_clones":0,"num_object_copies":234952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95987,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8998,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96014,"ondisk_log_size":96014,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1682472,"kb_used_data":933680,"kb_used_omap":2,"kb_used_meta":748605,"kb_avail":375804888,"statfs":{"total":386547056640,"available":384824205312,"internally_reserved":0,"allocated":956088320,"data_stored":955231608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":766571676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,0,6,11,22,13,0,0,0,3,0,2,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22568960,"num_objects":-2755,"num_object_clones":0,"num_object_copies":-11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008864"},"pg_stats":[{"pgid":"1.f","version":"40'5872","reported_seq":"6098","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.767720+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:24.767720+0000","last_peered":"2021-01-15T18:45:24.767720+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:24.767720+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5872,"ondisk_log_size":5872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29786112,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2370}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5916","reported_seq":"8840","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.773037+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:24.773037+0000","last_peered":"2021-01-15T18:45:24.773037+0000","last_clean":"2021-01-15T18:45:24.773037+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:24.773037+0000","last_undegraded":"2021-01-15T18:45:24.773037+0000","last_fullsized":"2021-01-15T18:45:24.773037+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5916,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29040806,"num_objects":3546,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5920","reported_seq":"6062","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:23.468866+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:23.468866+0000","last_peered":"2021-01-15T18:45:23.468866+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:23.468866+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5920,"ondisk_log_size":5920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29257835,"num_objects":3573,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5919,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2186}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'6085","reported_seq":"6295","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.555734+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:24.555734+0000","last_peered":"2021-01-15T18:45:24.555734+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:24.555734+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6085,"ondisk_log_size":6085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30367744,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6085,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6142","reported_seq":"6734","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.681834+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:24.681834+0000","last_peered":"2021-01-15T18:45:24.681834+0000","last_clean":"2021-01-15T18:45:24.681834+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:24.681834+0000","last_undegraded":"2021-01-15T18:45:24.681834+0000","last_fullsized":"2021-01-15T18:45:24.681834+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6142,"ondisk_log_size":6142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30195878,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5893","reported_seq":"6104","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.755174+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:24.755174+0000","last_peered":"2021-01-15T18:45:24.755174+0000","last_clean":"2021-01-15T18:45:24.755174+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:24.755174+0000","last_undegraded":"2021-01-15T18:45:24.755174+0000","last_fullsized":"2021-01-15T18:45:24.755174+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5893,"ondisk_log_size":5893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29761702,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5891,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'6003","reported_seq":"6219","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.863044+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:24.863044+0000","last_peered":"2021-01-15T18:45:24.863044+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:24.863044+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6003,"ondisk_log_size":6003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30490790,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6001,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6113","reported_seq":"6259","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:23.291924+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:23.291924+0000","last_peered":"2021-01-15T18:45:23.291924+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:23.291924+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6113,"ondisk_log_size":6113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30666835,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6112,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2326}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'6071","reported_seq":"6299","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.707773+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:24.707773+0000","last_peered":"2021-01-15T18:45:24.707773+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:24.707773+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6071,"ondisk_log_size":6071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29962323,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2313}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'6065","reported_seq":"9006","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.839429+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:24.839429+0000","last_peered":"2021-01-15T18:45:24.839429+0000","last_clean":"2021-01-15T18:45:24.839429+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:24.839429+0000","last_undegraded":"2021-01-15T18:45:24.839429+0000","last_fullsized":"2021-01-15T18:45:24.839429+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6065,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30249209,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6062,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5909","reported_seq":"6125","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.809191+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:24.809191+0000","last_peered":"2021-01-15T18:45:24.809191+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:24.809191+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5909,"ondisk_log_size":5909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29860006,"num_objects":3646,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5907,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5916","reported_seq":"8357","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.348750+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:24.348750+0000","last_peered":"2021-01-15T18:45:24.348750+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:24.348750+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5916,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29909158,"num_objects":3652,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5849","reported_seq":"8744","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:24.301862+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:24.301862+0000","last_peered":"2021-01-15T18:45:24.301862+0000","last_clean":"2021-01-15T18:45:24.301862+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:24.301862+0000","last_undegraded":"2021-01-15T18:45:24.301862+0000","last_fullsized":"2021-01-15T18:45:24.301862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5849,"ondisk_log_size":5849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29704358,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5847,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6143","reported_seq":"6261","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.126613+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:24.126613+0000","last_peered":"2021-01-15T18:45:24.126613+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:24.126613+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6143,"ondisk_log_size":6143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30662822,"num_objects":3744,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6236","reported_seq":"6456","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:24.878006+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:24.878006+0000","last_peered":"2021-01-15T18:45:24.878006+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:24.878006+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6236,"ondisk_log_size":6236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31179084,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6233,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5881","reported_seq":"8745","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:23.409114+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:23.409114+0000","last_peered":"2021-01-15T18:45:23.409114+0000","last_clean":"2021-01-15T18:45:23.409114+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:23.409114+0000","last_undegraded":"2021-01-15T18:45:23.409114+0000","last_fullsized":"2021-01-15T18:45:23.409114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5881,"ondisk_log_size":5881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29966502,"num_objects":3659,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":481061164,"num_objects":58738,"num_object_clones":0,"num_object_copies":234952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95987,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8998,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":953376768,"data_stored":953376768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96014,"ondisk_log_size":96014,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215175,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433048,"kb_used_data":235616,"kb_used_omap":0,"kb_used_meta":197375,"kb_avail":93938792,"statfs":{"total":96636764160,"available":96193323008,"internally_reserved":0,"allocated":241270784,"data_stored":241056606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202112052},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,2,2,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.872},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.96499999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":3.3319999999999999},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":3.427}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":24.062000000000001},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":10.772}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425296,"kb_used_data":236216,"kb_used_omap":0,"kb_used_meta":189056,"kb_avail":93946544,"statfs":{"total":96636764160,"available":96201261056,"internally_reserved":0,"allocated":241885184,"data_stored":241671006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":193593344},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,6,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":3.4409999999999998},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":2.8570000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":1.222},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":3.2869999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416684,"kb_used_data":235424,"kb_used_omap":0,"kb_used_meta":181183,"kb_avail":93955156,"statfs":{"total":96636764160,"available":96210079744,"internally_reserved":0,"allocated":241074176,"data_stored":240859998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":185531444},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,9,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":7.1349999999999998},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":6.8959999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.512},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":2.4020000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407444,"kb_used_data":226424,"kb_used_omap":0,"kb_used_meta":180991,"kb_avail":93964396,"statfs":{"total":96636764160,"available":96219541504,"internally_reserved":0,"allocated":231858176,"data_stored":231643998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":185334836},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.95699999999999996},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.752},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.85199999999999998}]}]}]}} 2021-01-15T18:45:30.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:31.549 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:31.579 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:31.601 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2021-01-15T18:45:30.543909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472770860,"num_objects":57726,"num_object_clones":0,"num_object_copies":230904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96999,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9160,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97026,"ondisk_log_size":97026,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1664440,"kb_used_data":919980,"kb_used_omap":2,"kb_used_meta":744381,"kb_avail":375822920,"statfs":{"total":386547056640,"available":384842670080,"internally_reserved":0,"allocated":942059520,"data_stored":941202808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":762246300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,7,10,16,19,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25649152,"num_objects":-3131,"num_object_clones":0,"num_object_copies":-12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010185"},"pg_stats":[{"pgid":"1.f","version":"40'5924","reported_seq":"6164","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.837752+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:29.837752+0000","last_peered":"2021-01-15T18:45:29.837752+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:29.837752+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5924,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29360128,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5979","reported_seq":"8903","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.863033+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:29.863033+0000","last_peered":"2021-01-15T18:45:29.863033+0000","last_clean":"2021-01-15T18:45:29.863033+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:29.863033+0000","last_undegraded":"2021-01-15T18:45:29.863033+0000","last_fullsized":"2021-01-15T18:45:29.863033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5979,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28524710,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5989","reported_seq":"6134","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:28.488444+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:28.488444+0000","last_peered":"2021-01-15T18:45:28.488444+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:28.488444+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5989,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28692587,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5988,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'6155","reported_seq":"6380","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.925946+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:29.925946+0000","last_peered":"2021-01-15T18:45:29.925946+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:29.925946+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6155,"ondisk_log_size":6155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29794304,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6155,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6210","reported_seq":"6801","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.837373+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:29.837373+0000","last_peered":"2021-01-15T18:45:29.837373+0000","last_clean":"2021-01-15T18:45:29.837373+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:29.837373+0000","last_undegraded":"2021-01-15T18:45:29.837373+0000","last_fullsized":"2021-01-15T18:45:29.837373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6210,"ondisk_log_size":6210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29638822,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5949","reported_seq":"6160","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.935415+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:29.935415+0000","last_peered":"2021-01-15T18:45:29.935415+0000","last_clean":"2021-01-15T18:45:29.935415+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:29.935415+0000","last_undegraded":"2021-01-15T18:45:29.935415+0000","last_fullsized":"2021-01-15T18:45:29.935415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5949,"ondisk_log_size":5949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29302950,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'6068","reported_seq":"6298","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.764558+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:29.764558+0000","last_peered":"2021-01-15T18:45:29.764558+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:29.764558+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6068,"ondisk_log_size":6068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29958310,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6186","reported_seq":"6340","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:28.536076+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:28.536076+0000","last_peered":"2021-01-15T18:45:28.536076+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:28.536076+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6186,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30068819,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'6129","reported_seq":"6368","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.914060+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:29.914060+0000","last_peered":"2021-01-15T18:45:29.914060+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:29.914060+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6129,"ondisk_log_size":6129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29487187,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'6125","reported_seq":"9066","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.569984+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:29.569984+0000","last_peered":"2021-01-15T18:45:29.569984+0000","last_clean":"2021-01-15T18:45:29.569984+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:29.569984+0000","last_undegraded":"2021-01-15T18:45:29.569984+0000","last_fullsized":"2021-01-15T18:45:29.569984+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6125,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29757689,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5973","reported_seq":"6197","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.888852+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:29.888852+0000","last_peered":"2021-01-15T18:45:29.888852+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:29.888852+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5973,"ondisk_log_size":5973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29335718,"num_objects":3582,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5969","reported_seq":"8636","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.395844+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:29.395844+0000","last_peered":"2021-01-15T18:45:29.395844+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:29.395844+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5969,"ondisk_log_size":5969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29474982,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1301,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5927","reported_seq":"8822","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.304358+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:29.304358+0000","last_peered":"2021-01-15T18:45:29.304358+0000","last_clean":"2021-01-15T18:45:29.304358+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:29.304358+0000","last_undegraded":"2021-01-15T18:45:29.304358+0000","last_fullsized":"2021-01-15T18:45:29.304358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5927,"ondisk_log_size":5927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6198","reported_seq":"6323","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.314167+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:29.314167+0000","last_peered":"2021-01-15T18:45:29.314167+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:29.314167+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6198,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30212262,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6310","reported_seq":"6546","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.858706+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:29.858706+0000","last_peered":"2021-01-15T18:45:29.858706+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:29.858706+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6310,"ondisk_log_size":6310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30572876,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5935","reported_seq":"8800","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:28.334323+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:28.334323+0000","last_peered":"2021-01-15T18:45:28.334323+0000","last_clean":"2021-01-15T18:45:28.334323+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:28.334323+0000","last_undegraded":"2021-01-15T18:45:28.334323+0000","last_fullsized":"2021-01-15T18:45:28.334323+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5935,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29524134,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":472770860,"num_objects":57726,"num_object_clones":0,"num_object_copies":230904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96999,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9160,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":939278336,"data_stored":939278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97026,"ondisk_log_size":97026,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422916,"kb_used_data":231648,"kb_used_omap":0,"kb_used_meta":191231,"kb_avail":93948924,"statfs":{"total":96636764160,"available":96203698176,"internally_reserved":0,"allocated":237207552,"data_stored":236993374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195820596},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,9,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.751},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.95999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.871},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.0640000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":2.641},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.429}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419220,"kb_used_data":232064,"kb_used_omap":0,"kb_used_meta":187136,"kb_avail":93952620,"statfs":{"total":96636764160,"available":96207482880,"internally_reserved":0,"allocated":237633536,"data_stored":237419358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":191627264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":1.103}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.877}]}]},{"osd":0,"up_from":22,"seq":94489280588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422940,"kb_used_data":231568,"kb_used_omap":0,"kb_used_meta":191359,"kb_avail":93948900,"statfs":{"total":96636764160,"available":96203673600,"internally_reserved":0,"allocated":237125632,"data_stored":236911454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195951668},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":1.764},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.101}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":6.1660000000000004},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.77400000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399364,"kb_used_data":224700,"kb_used_omap":0,"kb_used_meta":174655,"kb_avail":93972476,"statfs":{"total":96636764160,"available":96227815424,"internally_reserved":0,"allocated":230092800,"data_stored":229878622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":178846772},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.3570000000000002},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.7400000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.70499999999999996},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":9.5980000000000008},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":4.859}]}]}]}} 2021-01-15T18:45:31.603 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:32.784 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:32.784 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12247/230904 objects degraded (5.304%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":40,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57726,"data_bytes":472770860,"bytes_used":1704386560,"bytes_avail":384842670080,"bytes_total":386547056640,"degraded_objects":12247,"degraded_total":230904,"degraded_ratio":0.05303935834805807,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":172},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:32.837 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:32.837 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:33.460 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:33.474 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:33.517 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2021-01-15T18:45:32.544452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472770860,"num_objects":57726,"num_object_clones":0,"num_object_copies":230904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96999,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9160,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97026,"ondisk_log_size":97026,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1664440,"kb_used_data":919980,"kb_used_omap":2,"kb_used_meta":744381,"kb_avail":375822920,"statfs":{"total":386547056640,"available":384842670080,"internally_reserved":0,"allocated":942059520,"data_stored":941202808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":762246300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,7,10,16,19,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16990208,"num_objects":-2074,"num_object_clones":0,"num_object_copies":-8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006379"},"pg_stats":[{"pgid":"1.f","version":"40'5924","reported_seq":"6164","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.837752+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:29.837752+0000","last_peered":"2021-01-15T18:45:29.837752+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:29.837752+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5924,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29360128,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"40'5979","reported_seq":"8903","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.863033+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:29.863033+0000","last_peered":"2021-01-15T18:45:29.863033+0000","last_clean":"2021-01-15T18:45:29.863033+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:29.863033+0000","last_undegraded":"2021-01-15T18:45:29.863033+0000","last_fullsized":"2021-01-15T18:45:29.863033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5979,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28524710,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'5989","reported_seq":"6134","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:28.488444+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:28.488444+0000","last_peered":"2021-01-15T18:45:28.488444+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:28.488444+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5989,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28692587,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5988,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"40'6155","reported_seq":"6380","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.925946+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:29.925946+0000","last_peered":"2021-01-15T18:45:29.925946+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:29.925946+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6155,"ondisk_log_size":6155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29794304,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6155,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"40'6210","reported_seq":"6801","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.837373+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:29.837373+0000","last_peered":"2021-01-15T18:45:29.837373+0000","last_clean":"2021-01-15T18:45:29.837373+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:29.837373+0000","last_undegraded":"2021-01-15T18:45:29.837373+0000","last_fullsized":"2021-01-15T18:45:29.837373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6210,"ondisk_log_size":6210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29638822,"num_objects":3619,"num_object_clones":0,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"40'5949","reported_seq":"6160","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.935415+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:29.935415+0000","last_peered":"2021-01-15T18:45:29.935415+0000","last_clean":"2021-01-15T18:45:29.935415+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:29.935415+0000","last_undegraded":"2021-01-15T18:45:29.935415+0000","last_fullsized":"2021-01-15T18:45:29.935415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5949,"ondisk_log_size":5949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29302950,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"40'6068","reported_seq":"6298","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.764558+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:29.764558+0000","last_peered":"2021-01-15T18:45:29.764558+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:29.764558+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6068,"ondisk_log_size":6068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29958310,"num_objects":3658,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6186","reported_seq":"6340","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:28.536076+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:28.536076+0000","last_peered":"2021-01-15T18:45:28.536076+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:28.536076+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6186,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30068819,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"40'6129","reported_seq":"6368","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.914060+0000","last_change":"2021-01-15T18:40:32.752184+0000","last_active":"2021-01-15T18:45:29.914060+0000","last_peered":"2021-01-15T18:45:29.914060+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:29.914060+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6129,"ondisk_log_size":6129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29487187,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"40'6125","reported_seq":"9066","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.569984+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:29.569984+0000","last_peered":"2021-01-15T18:45:29.569984+0000","last_clean":"2021-01-15T18:45:29.569984+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:29.569984+0000","last_undegraded":"2021-01-15T18:45:29.569984+0000","last_fullsized":"2021-01-15T18:45:29.569984+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6125,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29757689,"num_objects":3634,"num_object_clones":0,"num_object_copies":14536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"40'5973","reported_seq":"6197","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.888852+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:29.888852+0000","last_peered":"2021-01-15T18:45:29.888852+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:29.888852+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5973,"ondisk_log_size":5973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29335718,"num_objects":3582,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'5969","reported_seq":"8636","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.395844+0000","last_change":"2021-01-15T18:44:22.931581+0000","last_active":"2021-01-15T18:45:29.395844+0000","last_peered":"2021-01-15T18:45:29.395844+0000","last_clean":"2021-01-15T18:39:34.164772+0000","last_became_active":"2021-01-15T18:40:32.765390+0000","last_became_peered":"2021-01-15T18:40:32.765390+0000","last_unstale":"2021-01-15T18:45:29.395844+0000","last_undegraded":"2021-01-15T18:40:32.526455+0000","last_fullsized":"2021-01-15T18:40:32.488627+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5969,"ondisk_log_size":5969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29474982,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1301,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"40'5927","reported_seq":"8822","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:29.304358+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:29.304358+0000","last_peered":"2021-01-15T18:45:29.304358+0000","last_clean":"2021-01-15T18:45:29.304358+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:29.304358+0000","last_undegraded":"2021-01-15T18:45:29.304358+0000","last_fullsized":"2021-01-15T18:45:29.304358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5927,"ondisk_log_size":5927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"40'6198","reported_seq":"6323","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.314167+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:29.314167+0000","last_peered":"2021-01-15T18:45:29.314167+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:29.314167+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6198,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30212262,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"40'6310","reported_seq":"6546","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:29.858706+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:29.858706+0000","last_peered":"2021-01-15T18:45:29.858706+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:29.858706+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6310,"ondisk_log_size":6310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30572876,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'5935","reported_seq":"8800","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:28.334323+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:28.334323+0000","last_peered":"2021-01-15T18:45:28.334323+0000","last_clean":"2021-01-15T18:45:28.334323+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:28.334323+0000","last_undegraded":"2021-01-15T18:45:28.334323+0000","last_fullsized":"2021-01-15T18:45:28.334323+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5935,"ondisk_log_size":5935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29524134,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":472770860,"num_objects":57726,"num_object_clones":0,"num_object_copies":230904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96999,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9160,"num_bytes_recovered":67641344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":939278336,"data_stored":939278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97026,"ondisk_log_size":97026,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215176,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422916,"kb_used_data":231648,"kb_used_omap":0,"kb_used_meta":191231,"kb_avail":93948924,"statfs":{"total":96636764160,"available":96203698176,"internally_reserved":0,"allocated":237207552,"data_stored":236993374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195820596},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,9,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.751},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.95999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.871},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.0640000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":2.641},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.429}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419220,"kb_used_data":232064,"kb_used_omap":0,"kb_used_meta":187136,"kb_avail":93952620,"statfs":{"total":96636764160,"available":96207482880,"internally_reserved":0,"allocated":237633536,"data_stored":237419358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":191627264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":1.103}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.877}]}]},{"osd":0,"up_from":22,"seq":94489280588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422940,"kb_used_data":231568,"kb_used_omap":0,"kb_used_meta":191359,"kb_avail":93948900,"statfs":{"total":96636764160,"available":96203673600,"internally_reserved":0,"allocated":237125632,"data_stored":236911454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195951668},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:44:32 2021","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":2.609,"15min":2.8290000000000002},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.4279999999999999,"5min":13.311,"15min":13.311},"last":1.764},{"interface":"front","average":{"1min":3.0910000000000002,"5min":3.7440000000000002,"15min":2.6429999999999998},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":14.942,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.101}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":6.1660000000000004},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.77400000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399364,"kb_used_data":224700,"kb_used_omap":0,"kb_used_meta":174655,"kb_avail":93972476,"statfs":{"total":96636764160,"available":96227815424,"internally_reserved":0,"allocated":230092800,"data_stored":229878622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":178846772},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.3570000000000002},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.7400000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.70499999999999996},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":9.5980000000000008},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":4.859}]}]}]}} 2021-01-15T18:45:36.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:37.191 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:37.193 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:37.204 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2021-01-15T18:45:36.548733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":464759084,"num_objects":56748,"num_object_clones":0,"num_object_copies":226992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97977,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9311,"num_bytes_recovered":68075520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98004,"ondisk_log_size":98004,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1662360,"kb_used_data":905596,"kb_used_omap":2,"kb_used_meta":756669,"kb_avail":375825000,"statfs":{"total":386547056640,"available":384844800000,"internally_reserved":0,"allocated":927330304,"data_stored":926459165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":774829212},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,16,13,8,5,1,2,5,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16302080,"num_objects":-1990,"num_object_clones":0,"num_object_copies":-7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1990,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005312"},"pg_stats":[{"pgid":"1.f","version":"42'5983","reported_seq":"6227","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.926123+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:34.926123+0000","last_peered":"2021-01-15T18:45:34.926123+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:34.926123+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5983,"ondisk_log_size":5983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28876800,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5983,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6037","reported_seq":"8963","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.879712+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:34.879712+0000","last_peered":"2021-01-15T18:45:34.879712+0000","last_clean":"2021-01-15T18:45:34.879712+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:34.879712+0000","last_undegraded":"2021-01-15T18:45:34.879712+0000","last_fullsized":"2021-01-15T18:45:34.879712+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6037,"ondisk_log_size":6037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28049574,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'6049","reported_seq":"6195","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:33.125689+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:33.125689+0000","last_peered":"2021-01-15T18:45:33.125689+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:33.125689+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6049,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28201067,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6048,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6210","reported_seq":"6446","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.801658+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:34.801658+0000","last_peered":"2021-01-15T18:45:34.801658+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:34.801658+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6210,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29343744,"num_objects":3582,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6272","reported_seq":"6866","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.983829+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:34.983829+0000","last_peered":"2021-01-15T18:45:34.983829+0000","last_clean":"2021-01-15T18:45:34.983829+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:34.983829+0000","last_undegraded":"2021-01-15T18:45:34.983829+0000","last_fullsized":"2021-01-15T18:45:34.983829+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6272,"ondisk_log_size":6272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29130918,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6001","reported_seq":"6214","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.799455+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:34.799455+0000","last_peered":"2021-01-15T18:45:34.799455+0000","last_clean":"2021-01-15T18:45:34.799455+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:34.799455+0000","last_undegraded":"2021-01-15T18:45:34.799455+0000","last_fullsized":"2021-01-15T18:45:34.799455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6001,"ondisk_log_size":6001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28876966,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5999,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"41'6121","reported_seq":"6365","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.796183+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:34.796183+0000","last_peered":"2021-01-15T18:45:34.796183+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:34.796183+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6121,"ondisk_log_size":6121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29524134,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6249","reported_seq":"6403","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:33.491031+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:33.491031+0000","last_peered":"2021-01-15T18:45:33.491031+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:33.491031+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6249,"ondisk_log_size":6249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29552723,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"41'6192","reported_seq":"6545","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.970417+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:34.970417+0000","last_peered":"2021-01-15T18:45:34.970417+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:34.970417+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6192,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28971091,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6189","reported_seq":"9132","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.892731+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:34.892731+0000","last_peered":"2021-01-15T18:45:34.892731+0000","last_clean":"2021-01-15T18:45:34.892731+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:34.892731+0000","last_undegraded":"2021-01-15T18:45:34.892731+0000","last_fullsized":"2021-01-15T18:45:34.892731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6189,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29233401,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6041","reported_seq":"6277","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.910810+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:34.910810+0000","last_peered":"2021-01-15T18:45:34.910810+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:34.910810+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6041,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28778662,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'6032","reported_seq":"8862","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:34.906257+0000","last_change":"2021-01-15T18:45:34.906257+0000","last_active":"2021-01-15T18:45:34.906257+0000","last_peered":"2021-01-15T18:45:34.906257+0000","last_clean":"2021-01-15T18:45:34.906257+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:34.906257+0000","last_undegraded":"2021-01-15T18:45:34.906257+0000","last_fullsized":"2021-01-15T18:45:34.906257+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6032,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"41'5991","reported_seq":"8887","reported_epoch":"41","state":"active+clean","last_fresh":"2021-01-15T18:45:34.202010+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:34.202010+0000","last_peered":"2021-01-15T18:45:34.202010+0000","last_clean":"2021-01-15T18:45:34.202010+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:34.202010+0000","last_undegraded":"2021-01-15T18:45:34.202010+0000","last_fullsized":"2021-01-15T18:45:34.202010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5991,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28541094,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5989,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"41'6260","reported_seq":"6386","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.085699+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.085699+0000","last_peered":"2021-01-15T18:45:34.085699+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.085699+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6260,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29704358,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"41'6373","reported_seq":"6623","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.800660+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:34.800660+0000","last_peered":"2021-01-15T18:45:34.800660+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:34.800660+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6373,"ondisk_log_size":6373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30056780,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6370,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'6004","reported_seq":"8869","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:33.556278+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:33.556278+0000","last_peered":"2021-01-15T18:45:33.556278+0000","last_clean":"2021-01-15T18:45:33.556278+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:33.556278+0000","last_undegraded":"2021-01-15T18:45:33.556278+0000","last_fullsized":"2021-01-15T18:45:33.556278+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":464759084,"num_objects":56748,"num_object_clones":0,"num_object_copies":226992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97977,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9311,"num_bytes_recovered":68075520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925343744,"data_stored":925343744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98004,"ondisk_log_size":98004,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428964,"kb_used_data":227400,"kb_used_omap":0,"kb_used_meta":201535,"kb_avail":93942876,"statfs":{"total":96636764160,"available":96197505024,"internally_reserved":0,"allocated":232857600,"data_stored":232638613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":206371892},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,7,1,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.1399999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.006},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.1439999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.86899999999999999},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":23.154}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425292,"kb_used_data":227932,"kb_used_omap":0,"kb_used_meta":197312,"kb_avail":93946548,"statfs":{"total":96636764160,"available":96201265152,"internally_reserved":0,"allocated":233402368,"data_stored":233188190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202047488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":6.0640000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":5.9290000000000003},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":6.0119999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":5.3540000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416744,"kb_used_data":227360,"kb_used_omap":0,"kb_used_meta":189375,"kb_avail":93955096,"statfs":{"total":96636764160,"available":96210018304,"internally_reserved":0,"allocated":232816640,"data_stored":232597653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":193920052},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":35.406999999999996},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":17.222999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.72499999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.70399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":5.5309999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391360,"kb_used_data":222904,"kb_used_omap":0,"kb_used_meta":168447,"kb_avail":93980480,"statfs":{"total":96636764160,"available":96236011520,"internally_reserved":0,"allocated":228253696,"data_stored":228034709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":172489780},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,0,2,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.762},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.1499999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.7890000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.056}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.325},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.234}]}]}]}} 2021-01-15T18:45:37.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:37.524 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:37.525 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12096/226992 objects degraded (5.329%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":388,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56748,"data_bytes":464759084,"bytes_used":1702256640,"bytes_avail":384844800000,"bytes_total":386547056640,"degraded_objects":12096,"degraded_total":226992,"degraded_ratio":0.053288221611334313,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":54235,"recovering_keys_per_sec":0,"num_objects_recovered":313,"num_bytes_recovered":434176,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":248},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:37.540 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:37.540 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:37.842 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:37.844 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:37.856 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2021-01-15T18:45:36.548733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":464759084,"num_objects":56748,"num_object_clones":0,"num_object_copies":226992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97977,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9311,"num_bytes_recovered":68075520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98004,"ondisk_log_size":98004,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1662360,"kb_used_data":905596,"kb_used_omap":2,"kb_used_meta":756669,"kb_avail":375825000,"statfs":{"total":386547056640,"available":384844800000,"internally_reserved":0,"allocated":927330304,"data_stored":926459165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":774829212},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,16,13,8,5,1,2,5,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16302080,"num_objects":-1990,"num_object_clones":0,"num_object_copies":-7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1990,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005312"},"pg_stats":[{"pgid":"1.f","version":"42'5983","reported_seq":"6227","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.926123+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:34.926123+0000","last_peered":"2021-01-15T18:45:34.926123+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:34.926123+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5983,"ondisk_log_size":5983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28876800,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5983,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2267}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6037","reported_seq":"8963","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.879712+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:34.879712+0000","last_peered":"2021-01-15T18:45:34.879712+0000","last_clean":"2021-01-15T18:45:34.879712+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:34.879712+0000","last_undegraded":"2021-01-15T18:45:34.879712+0000","last_fullsized":"2021-01-15T18:45:34.879712+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6037,"ondisk_log_size":6037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28049574,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"40'6049","reported_seq":"6195","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:33.125689+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:33.125689+0000","last_peered":"2021-01-15T18:45:33.125689+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:33.125689+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6049,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28201067,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6048,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2059}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6210","reported_seq":"6446","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.801658+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:34.801658+0000","last_peered":"2021-01-15T18:45:34.801658+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:34.801658+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6210,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29343744,"num_objects":3582,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6272","reported_seq":"6866","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.983829+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:34.983829+0000","last_peered":"2021-01-15T18:45:34.983829+0000","last_clean":"2021-01-15T18:45:34.983829+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:34.983829+0000","last_undegraded":"2021-01-15T18:45:34.983829+0000","last_fullsized":"2021-01-15T18:45:34.983829+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6272,"ondisk_log_size":6272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29130918,"num_objects":3557,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6001","reported_seq":"6214","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.799455+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:34.799455+0000","last_peered":"2021-01-15T18:45:34.799455+0000","last_clean":"2021-01-15T18:45:34.799455+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:34.799455+0000","last_undegraded":"2021-01-15T18:45:34.799455+0000","last_fullsized":"2021-01-15T18:45:34.799455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6001,"ondisk_log_size":6001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28876966,"num_objects":3526,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5999,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"41'6121","reported_seq":"6365","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.796183+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:34.796183+0000","last_peered":"2021-01-15T18:45:34.796183+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:34.796183+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6121,"ondisk_log_size":6121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29524134,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2305}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"40'6249","reported_seq":"6403","reported_epoch":"40","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:33.491031+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:33.491031+0000","last_peered":"2021-01-15T18:45:33.491031+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:33.491031+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6249,"ondisk_log_size":6249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29552723,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"41'6192","reported_seq":"6545","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.970417+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:34.970417+0000","last_peered":"2021-01-15T18:45:34.970417+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:34.970417+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6192,"ondisk_log_size":6192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28971091,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":434176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6189","reported_seq":"9132","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:34.892731+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:34.892731+0000","last_peered":"2021-01-15T18:45:34.892731+0000","last_clean":"2021-01-15T18:45:34.892731+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:34.892731+0000","last_undegraded":"2021-01-15T18:45:34.892731+0000","last_fullsized":"2021-01-15T18:45:34.892731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6189,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29233401,"num_objects":3570,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6041","reported_seq":"6277","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.910810+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:34.910810+0000","last_peered":"2021-01-15T18:45:34.910810+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:34.910810+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6041,"ondisk_log_size":6041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28778662,"num_objects":3514,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'6032","reported_seq":"8862","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:34.906257+0000","last_change":"2021-01-15T18:45:34.906257+0000","last_active":"2021-01-15T18:45:34.906257+0000","last_peered":"2021-01-15T18:45:34.906257+0000","last_clean":"2021-01-15T18:45:34.906257+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:34.906257+0000","last_undegraded":"2021-01-15T18:45:34.906257+0000","last_fullsized":"2021-01-15T18:45:34.906257+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6032,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"41'5991","reported_seq":"8887","reported_epoch":"41","state":"active+clean","last_fresh":"2021-01-15T18:45:34.202010+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:34.202010+0000","last_peered":"2021-01-15T18:45:34.202010+0000","last_clean":"2021-01-15T18:45:34.202010+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:34.202010+0000","last_undegraded":"2021-01-15T18:45:34.202010+0000","last_fullsized":"2021-01-15T18:45:34.202010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5991,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28541094,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5989,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"41'6260","reported_seq":"6386","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.085699+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.085699+0000","last_peered":"2021-01-15T18:45:34.085699+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.085699+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6260,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29704358,"num_objects":3627,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"41'6373","reported_seq":"6623","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.800660+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:34.800660+0000","last_peered":"2021-01-15T18:45:34.800660+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:34.800660+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6373,"ondisk_log_size":6373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30056780,"num_objects":3671,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6370,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"40'6004","reported_seq":"8869","reported_epoch":"40","state":"active+clean","last_fresh":"2021-01-15T18:45:33.556278+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:33.556278+0000","last_peered":"2021-01-15T18:45:33.556278+0000","last_clean":"2021-01-15T18:45:33.556278+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:33.556278+0000","last_undegraded":"2021-01-15T18:45:33.556278+0000","last_fullsized":"2021-01-15T18:45:33.556278+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":464759084,"num_objects":56748,"num_object_clones":0,"num_object_copies":226992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97977,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9311,"num_bytes_recovered":68075520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925343744,"data_stored":925343744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98004,"ondisk_log_size":98004,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215177,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428964,"kb_used_data":227400,"kb_used_omap":0,"kb_used_meta":201535,"kb_avail":93942876,"statfs":{"total":96636764160,"available":96197505024,"internally_reserved":0,"allocated":232857600,"data_stored":232638613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":206371892},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,7,1,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.1399999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:44:35 2021","interfaces":[{"interface":"back","average":{"1min":4.9960000000000004,"5min":4.0970000000000004,"15min":3.4129999999999998},"min":{"1min":0.58599999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":30.129000000000001,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.006},{"interface":"front","average":{"1min":4.4660000000000002,"5min":3.3159999999999998,"15min":2.3980000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":24.998999999999999,"5min":24.998999999999999,"15min":24.998999999999999},"last":1.1439999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.86899999999999999},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":23.154}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425292,"kb_used_data":227932,"kb_used_omap":0,"kb_used_meta":197312,"kb_avail":93946548,"statfs":{"total":96636764160,"available":96201265152,"internally_reserved":0,"allocated":233402368,"data_stored":233188190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202047488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":6.0640000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:44:33 2021","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":2.794,"15min":2.4550000000000001},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":20.885000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":5.9290000000000003},{"interface":"front","average":{"1min":3.0299999999999998,"5min":2.1989999999999998,"15min":2.1120000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":20.98,"5min":20.98,"15min":20.98},"last":6.0119999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":5.3540000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416744,"kb_used_data":227360,"kb_used_omap":0,"kb_used_meta":189375,"kb_avail":93955096,"statfs":{"total":96636764160,"available":96210018304,"internally_reserved":0,"allocated":232816640,"data_stored":232597653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":193920052},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,1,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":35.406999999999996},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":17.222999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.72499999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.70399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":5.9020000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":5.5309999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391360,"kb_used_data":222904,"kb_used_omap":0,"kb_used_meta":168447,"kb_avail":93980480,"statfs":{"total":96636764160,"available":96236011520,"internally_reserved":0,"allocated":228253696,"data_stored":228034709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":172489780},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,0,2,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.762},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.1499999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":3.7890000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":1.056}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.325},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.234}]}]}]}} 2021-01-15T18:45:40.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:41.241 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:41.242 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:41.255 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2021-01-15T18:45:40.549513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650100,"kb_used_data":902996,"kb_used_omap":2,"kb_used_meta":747069,"kb_avail":375837260,"statfs":{"total":386547056640,"available":384857354240,"internally_reserved":0,"allocated":924667904,"data_stored":923772636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":764998812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,5,41],"upper_bound":16384},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9142272,"num_objects":-1116,"num_object_clones":0,"num_object_copies":-4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005060"},"pg_stats":[{"pgid":"1.f","version":"42'5986","reported_seq":"6230","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.455939+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:35.455939+0000","last_peered":"2021-01-15T18:45:35.455939+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:35.455939+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5986,"ondisk_log_size":5986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6040","reported_seq":"8966","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.534705+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:35.534705+0000","last_peered":"2021-01-15T18:45:35.534705+0000","last_clean":"2021-01-15T18:45:35.534705+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:35.534705+0000","last_undegraded":"2021-01-15T18:45:35.534705+0000","last_fullsized":"2021-01-15T18:45:35.534705+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6040,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28024998,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6065","reported_seq":"6213","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.360450+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:35.360450+0000","last_peered":"2021-01-15T18:45:35.360450+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:35.360450+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6065,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28069995,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6064,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6213","reported_seq":"6449","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.078243+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:35.078243+0000","last_peered":"2021-01-15T18:45:35.078243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:35.078243+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6213,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29319168,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6280","reported_seq":"6874","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.446566+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:35.446566+0000","last_peered":"2021-01-15T18:45:35.446566+0000","last_clean":"2021-01-15T18:45:35.446566+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:35.446566+0000","last_undegraded":"2021-01-15T18:45:35.446566+0000","last_fullsized":"2021-01-15T18:45:35.446566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6280,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6009","reported_seq":"6222","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.516641+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:35.516641+0000","last_peered":"2021-01-15T18:45:35.516641+0000","last_clean":"2021-01-15T18:45:35.516641+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:35.516641+0000","last_undegraded":"2021-01-15T18:45:35.516641+0000","last_fullsized":"2021-01-15T18:45:35.516641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6009,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28811430,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6128","reported_seq":"6372","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.191742+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:35.191742+0000","last_peered":"2021-01-15T18:45:35.191742+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:35.191742+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6128,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29466790,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6268","reported_seq":"6424","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.125092+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:35.125092+0000","last_peered":"2021-01-15T18:45:35.125092+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:35.125092+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6268,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29397075,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6196","reported_seq":"7569","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:39.972618+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:39.972618+0000","last_peered":"2021-01-15T18:45:39.972618+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:39.972618+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28938323,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":663,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6194","reported_seq":"9137","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.346113+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:35.346113+0000","last_peered":"2021-01-15T18:45:35.346113+0000","last_clean":"2021-01-15T18:45:35.346113+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:35.346113+0000","last_undegraded":"2021-01-15T18:45:35.346113+0000","last_fullsized":"2021-01-15T18:45:35.346113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6194,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29192441,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6051","reported_seq":"6287","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.591516+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:35.591516+0000","last_peered":"2021-01-15T18:45:35.591516+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:35.591516+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6051,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28696742,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'6032","reported_seq":"8862","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:34.906257+0000","last_change":"2021-01-15T18:45:34.906257+0000","last_active":"2021-01-15T18:45:34.906257+0000","last_peered":"2021-01-15T18:45:34.906257+0000","last_clean":"2021-01-15T18:45:34.906257+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:34.906257+0000","last_undegraded":"2021-01-15T18:45:34.906257+0000","last_fullsized":"2021-01-15T18:45:34.906257+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6032,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6004","reported_seq":"8901","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.427489+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:35.427489+0000","last_peered":"2021-01-15T18:45:35.427489+0000","last_clean":"2021-01-15T18:45:35.427489+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:35.427489+0000","last_undegraded":"2021-01-15T18:45:35.427489+0000","last_fullsized":"2021-01-15T18:45:35.427489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28434598,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6274","reported_seq":"6409","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.921040+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.921040+0000","last_peered":"2021-01-15T18:45:34.921040+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.921040+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6274,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29589670,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6375","reported_seq":"6625","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.204624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:35.204624+0000","last_peered":"2021-01-15T18:45:35.204624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:35.204624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6375,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30040396,"num_objects":3669,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6372,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6027","reported_seq":"8894","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.435540+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:35.435540+0000","last_peered":"2021-01-15T18:45:35.435540+0000","last_clean":"2021-01-15T18:45:35.435540+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:35.435540+0000","last_undegraded":"2021-01-15T18:45:35.435540+0000","last_fullsized":"2021-01-15T18:45:35.435540+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6027,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28770470,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":923258880,"data_stored":923258880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416380,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":189695,"kb_avail":93955460,"statfs":{"total":96636764160,"available":96210391040,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194247732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78000000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.48899999999999999},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412476,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":185792,"kb_avail":93959364,"statfs":{"total":96636764160,"available":96214388736,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":190251008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.626},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.64800000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420540,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":193855,"kb_avail":93951300,"statfs":{"total":96636764160,"available":96206131200,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198507572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.74399999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400704,"kb_used_data":222968,"kb_used_omap":0,"kb_used_meta":177727,"kb_avail":93971136,"statfs":{"total":96636764160,"available":96226443264,"internally_reserved":0,"allocated":228319232,"data_stored":228095415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181992500},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,2,39],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.498}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.61099999999999999}]}]}]}} 2021-01-15T18:45:41.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:41.615 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:41.615 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11582/226440 objects degraded (5.115%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56610,"data_bytes":463628588,"bytes_used":1689702400,"bytes_avail":384857354240,"bytes_total":386547056640,"degraded_objects":11582,"degraded_total":226440,"degraded_ratio":0.051148207030559972,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":133035,"recovering_keys_per_sec":0,"num_objects_recovered":665,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":139},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:41.628 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:41.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2021-01-15T18:45:40.549513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650100,"kb_used_data":902996,"kb_used_omap":2,"kb_used_meta":747069,"kb_avail":375837260,"statfs":{"total":386547056640,"available":384857354240,"internally_reserved":0,"allocated":924667904,"data_stored":923772636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":764998812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,5,41],"upper_bound":16384},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9142272,"num_objects":-1116,"num_object_clones":0,"num_object_copies":-4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005060"},"pg_stats":[{"pgid":"1.f","version":"42'5986","reported_seq":"6230","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.455939+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:35.455939+0000","last_peered":"2021-01-15T18:45:35.455939+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:35.455939+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5986,"ondisk_log_size":5986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6040","reported_seq":"8966","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.534705+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:35.534705+0000","last_peered":"2021-01-15T18:45:35.534705+0000","last_clean":"2021-01-15T18:45:35.534705+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:35.534705+0000","last_undegraded":"2021-01-15T18:45:35.534705+0000","last_fullsized":"2021-01-15T18:45:35.534705+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6040,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28024998,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6065","reported_seq":"6213","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.360450+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:35.360450+0000","last_peered":"2021-01-15T18:45:35.360450+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:35.360450+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6065,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28069995,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6064,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6213","reported_seq":"6449","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.078243+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:35.078243+0000","last_peered":"2021-01-15T18:45:35.078243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:35.078243+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6213,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29319168,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6280","reported_seq":"6874","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.446566+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:35.446566+0000","last_peered":"2021-01-15T18:45:35.446566+0000","last_clean":"2021-01-15T18:45:35.446566+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:35.446566+0000","last_undegraded":"2021-01-15T18:45:35.446566+0000","last_fullsized":"2021-01-15T18:45:35.446566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6280,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6009","reported_seq":"6222","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.516641+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:35.516641+0000","last_peered":"2021-01-15T18:45:35.516641+0000","last_clean":"2021-01-15T18:45:35.516641+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:35.516641+0000","last_undegraded":"2021-01-15T18:45:35.516641+0000","last_fullsized":"2021-01-15T18:45:35.516641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6009,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28811430,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6128","reported_seq":"6372","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.191742+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:35.191742+0000","last_peered":"2021-01-15T18:45:35.191742+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:35.191742+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6128,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29466790,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6268","reported_seq":"6424","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.125092+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:35.125092+0000","last_peered":"2021-01-15T18:45:35.125092+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:35.125092+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6268,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29397075,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6196","reported_seq":"7569","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:39.972618+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:39.972618+0000","last_peered":"2021-01-15T18:45:39.972618+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:39.972618+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28938323,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":663,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6194","reported_seq":"9137","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.346113+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:35.346113+0000","last_peered":"2021-01-15T18:45:35.346113+0000","last_clean":"2021-01-15T18:45:35.346113+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:35.346113+0000","last_undegraded":"2021-01-15T18:45:35.346113+0000","last_fullsized":"2021-01-15T18:45:35.346113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6194,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29192441,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6051","reported_seq":"6287","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.591516+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:35.591516+0000","last_peered":"2021-01-15T18:45:35.591516+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:35.591516+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6051,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28696742,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'6032","reported_seq":"8862","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:34.906257+0000","last_change":"2021-01-15T18:45:34.906257+0000","last_active":"2021-01-15T18:45:34.906257+0000","last_peered":"2021-01-15T18:45:34.906257+0000","last_clean":"2021-01-15T18:45:34.906257+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:34.906257+0000","last_undegraded":"2021-01-15T18:45:34.906257+0000","last_fullsized":"2021-01-15T18:45:34.906257+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6032,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6004","reported_seq":"8901","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.427489+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:35.427489+0000","last_peered":"2021-01-15T18:45:35.427489+0000","last_clean":"2021-01-15T18:45:35.427489+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:35.427489+0000","last_undegraded":"2021-01-15T18:45:35.427489+0000","last_fullsized":"2021-01-15T18:45:35.427489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28434598,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6274","reported_seq":"6409","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.921040+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.921040+0000","last_peered":"2021-01-15T18:45:34.921040+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.921040+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6274,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29589670,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6375","reported_seq":"6625","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.204624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:35.204624+0000","last_peered":"2021-01-15T18:45:35.204624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:35.204624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6375,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30040396,"num_objects":3669,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6372,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6027","reported_seq":"8894","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.435540+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:35.435540+0000","last_peered":"2021-01-15T18:45:35.435540+0000","last_clean":"2021-01-15T18:45:35.435540+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:35.435540+0000","last_undegraded":"2021-01-15T18:45:35.435540+0000","last_fullsized":"2021-01-15T18:45:35.435540+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6027,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28770470,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":923258880,"data_stored":923258880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416380,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":189695,"kb_avail":93955460,"statfs":{"total":96636764160,"available":96210391040,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194247732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78000000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.48899999999999999},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412476,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":185792,"kb_avail":93959364,"statfs":{"total":96636764160,"available":96214388736,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":190251008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.626},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.64800000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420540,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":193855,"kb_avail":93951300,"statfs":{"total":96636764160,"available":96206131200,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198507572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.74399999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400704,"kb_used_data":222968,"kb_used_omap":0,"kb_used_meta":177727,"kb_avail":93971136,"statfs":{"total":96636764160,"available":96226443264,"internally_reserved":0,"allocated":228319232,"data_stored":228095415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181992500},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,2,39],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.498}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.61099999999999999}]}]}]}} 2021-01-15T18:45:44.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:45.935 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:45.977 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:46.006 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2021-01-15T18:45:44.550287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650100,"kb_used_data":902996,"kb_used_omap":2,"kb_used_meta":747069,"kb_avail":375837260,"statfs":{"total":386547056640,"available":384857354240,"internally_reserved":0,"allocated":924667904,"data_stored":923772636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":764998812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,5,39,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6537216,"num_objects":-798,"num_object_clones":0,"num_object_copies":-3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002019"},"pg_stats":[{"pgid":"1.f","version":"42'5986","reported_seq":"6230","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.455939+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:35.455939+0000","last_peered":"2021-01-15T18:45:35.455939+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:35.455939+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5986,"ondisk_log_size":5986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6040","reported_seq":"8966","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.534705+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:35.534705+0000","last_peered":"2021-01-15T18:45:35.534705+0000","last_clean":"2021-01-15T18:45:35.534705+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:35.534705+0000","last_undegraded":"2021-01-15T18:45:35.534705+0000","last_fullsized":"2021-01-15T18:45:35.534705+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6040,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28024998,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6065","reported_seq":"6213","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.360450+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:35.360450+0000","last_peered":"2021-01-15T18:45:35.360450+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:35.360450+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6065,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28069995,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6064,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6213","reported_seq":"6449","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.078243+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:35.078243+0000","last_peered":"2021-01-15T18:45:35.078243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:35.078243+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6213,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29319168,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6280","reported_seq":"6874","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.446566+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:35.446566+0000","last_peered":"2021-01-15T18:45:35.446566+0000","last_clean":"2021-01-15T18:45:35.446566+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:35.446566+0000","last_undegraded":"2021-01-15T18:45:35.446566+0000","last_fullsized":"2021-01-15T18:45:35.446566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6280,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6009","reported_seq":"6222","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.516641+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:35.516641+0000","last_peered":"2021-01-15T18:45:35.516641+0000","last_clean":"2021-01-15T18:45:35.516641+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:35.516641+0000","last_undegraded":"2021-01-15T18:45:35.516641+0000","last_fullsized":"2021-01-15T18:45:35.516641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6009,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28811430,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6128","reported_seq":"6372","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.191742+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:35.191742+0000","last_peered":"2021-01-15T18:45:35.191742+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:35.191742+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6128,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29466790,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6268","reported_seq":"6424","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.125092+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:35.125092+0000","last_peered":"2021-01-15T18:45:35.125092+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:35.125092+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6268,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29397075,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6196","reported_seq":"7569","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:39.972618+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:39.972618+0000","last_peered":"2021-01-15T18:45:39.972618+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:39.972618+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28938323,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":663,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2758}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6194","reported_seq":"9137","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.346113+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:35.346113+0000","last_peered":"2021-01-15T18:45:35.346113+0000","last_clean":"2021-01-15T18:45:35.346113+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:35.346113+0000","last_undegraded":"2021-01-15T18:45:35.346113+0000","last_fullsized":"2021-01-15T18:45:35.346113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6194,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29192441,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6051","reported_seq":"6287","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.591516+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:35.591516+0000","last_peered":"2021-01-15T18:45:35.591516+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:35.591516+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6051,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28696742,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"40'6032","reported_seq":"8862","reported_epoch":"42","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:34.906257+0000","last_change":"2021-01-15T18:45:34.906257+0000","last_active":"2021-01-15T18:45:34.906257+0000","last_peered":"2021-01-15T18:45:34.906257+0000","last_clean":"2021-01-15T18:45:34.906257+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:34.906257+0000","last_undegraded":"2021-01-15T18:45:34.906257+0000","last_fullsized":"2021-01-15T18:45:34.906257+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6032,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28958886,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6004","reported_seq":"8901","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.427489+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:35.427489+0000","last_peered":"2021-01-15T18:45:35.427489+0000","last_clean":"2021-01-15T18:45:35.427489+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:35.427489+0000","last_undegraded":"2021-01-15T18:45:35.427489+0000","last_fullsized":"2021-01-15T18:45:35.427489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28434598,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6274","reported_seq":"6409","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.921040+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.921040+0000","last_peered":"2021-01-15T18:45:34.921040+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.921040+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6274,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29589670,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6375","reported_seq":"6625","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.204624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:35.204624+0000","last_peered":"2021-01-15T18:45:35.204624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:35.204624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6375,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30040396,"num_objects":3669,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6372,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6027","reported_seq":"8894","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.435540+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:35.435540+0000","last_peered":"2021-01-15T18:45:35.435540+0000","last_clean":"2021-01-15T18:45:35.435540+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:35.435540+0000","last_undegraded":"2021-01-15T18:45:35.435540+0000","last_fullsized":"2021-01-15T18:45:35.435540+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6027,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28770470,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463628588,"num_objects":56610,"num_object_clones":0,"num_object_copies":226440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98115,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9825,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":923258880,"data_stored":923258880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98142,"ondisk_log_size":98142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416380,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":189695,"kb_avail":93955460,"statfs":{"total":96636764160,"available":96210391040,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194247732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78000000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.48899999999999999},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412476,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":185792,"kb_avail":93959364,"statfs":{"total":96636764160,"available":96214388736,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":190251008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.62},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.75900000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420540,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":193855,"kb_avail":93951300,"statfs":{"total":96636764160,"available":96206131200,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198507572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.66400000000000003},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.75800000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400704,"kb_used_data":222968,"kb_used_omap":0,"kb_used_meta":177727,"kb_avail":93971136,"statfs":{"total":96636764160,"available":96226443264,"internally_reserved":0,"allocated":228319232,"data_stored":228095415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181992500},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,2,39],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":5.0670000000000002,"5min":4.5229999999999997,"15min":3.3050000000000002},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.029,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":3.2450000000000001,"5min":2.641,"15min":2.1150000000000002},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":21.649999999999999,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.498}]},{"osd":2,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.512,"5min":2.9950000000000001,"15min":2.8149999999999999},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":21.791,"5min":21.791,"15min":21.791},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1539999999999999,"15min":1.8200000000000001},"min":{"1min":0.57899999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":12.247,"5min":19.381,"15min":19.381},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:40 2021","interfaces":[{"interface":"back","average":{"1min":3.9670000000000001,"5min":3.8620000000000001,"15min":4.0819999999999999},"min":{"1min":0.66500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.216000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":2.5449999999999999,"5min":2.5030000000000001,"15min":2.589},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.8499999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.61099999999999999}]}]}]}} 2021-01-15T18:45:46.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:46.818 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:46.819 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11582/226440 objects degraded (5.115%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":397,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56610,"data_bytes":463628588,"bytes_used":1689702400,"bytes_avail":384857354240,"bytes_total":386547056640,"degraded_objects":11582,"degraded_total":226440,"degraded_ratio":0.051148207030559972,"recovering_objects_per_sec":66,"recovering_bytes_per_sec":106474,"recovering_keys_per_sec":0,"num_objects_recovered":665,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":79},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:46.867 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:46.868 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:47.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:47.912 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:47.955 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2021-01-15T18:45:46.550793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463481132,"num_objects":56592,"num_object_clones":0,"num_object_copies":226368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56592,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98133,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10472,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98160,"ondisk_log_size":98160,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1646836,"kb_used_data":902996,"kb_used_omap":2,"kb_used_meta":743805,"kb_avail":375840524,"statfs":{"total":386547056640,"available":384860696576,"internally_reserved":0,"allocated":924667904,"data_stored":923772636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":761656476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,46],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6684672,"num_objects":-816,"num_object_clones":0,"num_object_copies":-3264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":816,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.772079"},"pg_stats":[{"pgid":"1.f","version":"42'5986","reported_seq":"6230","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.455939+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:35.455939+0000","last_peered":"2021-01-15T18:45:35.455939+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:35.455939+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":5986,"ondisk_log_size":5986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28852224,"num_objects":3522,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6040","reported_seq":"8966","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.534705+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:35.534705+0000","last_peered":"2021-01-15T18:45:35.534705+0000","last_clean":"2021-01-15T18:45:35.534705+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:35.534705+0000","last_undegraded":"2021-01-15T18:45:35.534705+0000","last_fullsized":"2021-01-15T18:45:35.534705+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6040,"ondisk_log_size":6040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28024998,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6065","reported_seq":"6213","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.360450+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:35.360450+0000","last_peered":"2021-01-15T18:45:35.360450+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:35.360450+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6065,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28069995,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6064,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6213","reported_seq":"6449","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.078243+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:35.078243+0000","last_peered":"2021-01-15T18:45:35.078243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:35.078243+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6213,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29319168,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2239}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6280","reported_seq":"6874","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.446566+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:35.446566+0000","last_peered":"2021-01-15T18:45:35.446566+0000","last_clean":"2021-01-15T18:45:35.446566+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:35.446566+0000","last_undegraded":"2021-01-15T18:45:35.446566+0000","last_fullsized":"2021-01-15T18:45:35.446566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6280,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29065382,"num_objects":3549,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6010","reported_seq":"6223","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.648912+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:35.648912+0000","last_peered":"2021-01-15T18:45:35.648912+0000","last_clean":"2021-01-15T18:45:35.648912+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:35.648912+0000","last_undegraded":"2021-01-15T18:45:35.648912+0000","last_fullsized":"2021-01-15T18:45:35.648912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6010,"ondisk_log_size":6010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28803238,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6008,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6128","reported_seq":"6372","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.191742+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:35.191742+0000","last_peered":"2021-01-15T18:45:35.191742+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:35.191742+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6128,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29466790,"num_objects":3598,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2298}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6268","reported_seq":"6424","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.125092+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:35.125092+0000","last_peered":"2021-01-15T18:45:35.125092+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:35.125092+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6268,"ondisk_log_size":6268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29397075,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6198","reported_seq":"8863","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:44.973123+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:44.973123+0000","last_peered":"2021-01-15T18:45:44.973123+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:44.973123+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6198,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28921939,"num_objects":3531,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1310,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6194","reported_seq":"9137","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.346113+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:35.346113+0000","last_peered":"2021-01-15T18:45:35.346113+0000","last_clean":"2021-01-15T18:45:35.346113+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:35.346113+0000","last_undegraded":"2021-01-15T18:45:35.346113+0000","last_fullsized":"2021-01-15T18:45:35.346113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6194,"ondisk_log_size":6194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29192441,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6051","reported_seq":"6287","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.591516+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:35.591516+0000","last_peered":"2021-01-15T18:45:35.591516+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:35.591516+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6051,"ondisk_log_size":6051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28696742,"num_objects":3504,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"42'6047","reported_seq":"8878","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:44.973944+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:44.973944+0000","last_peered":"2021-01-15T18:45:44.973944+0000","last_clean":"2021-01-15T18:45:44.973944+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:44.973944+0000","last_undegraded":"2021-01-15T18:45:44.973944+0000","last_fullsized":"2021-01-15T18:45:44.973944+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6047,"ondisk_log_size":6047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28836006,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6045,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6004","reported_seq":"8901","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.427489+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:35.427489+0000","last_peered":"2021-01-15T18:45:35.427489+0000","last_clean":"2021-01-15T18:45:35.427489+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:35.427489+0000","last_undegraded":"2021-01-15T18:45:35.427489+0000","last_fullsized":"2021-01-15T18:45:35.427489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6004,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28434598,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6002,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6274","reported_seq":"6409","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:34.921040+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:34.921040+0000","last_peered":"2021-01-15T18:45:34.921040+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:34.921040+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6274,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29589670,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2203}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6375","reported_seq":"6625","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:35.204624+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:35.204624+0000","last_peered":"2021-01-15T18:45:35.204624+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:35.204624+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6375,"ondisk_log_size":6375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30040396,"num_objects":3669,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6372,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2275}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6027","reported_seq":"8894","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:35.435540+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:35.435540+0000","last_peered":"2021-01-15T18:45:35.435540+0000","last_clean":"2021-01-15T18:45:35.435540+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:35.435540+0000","last_undegraded":"2021-01-15T18:45:35.435540+0000","last_fullsized":"2021-01-15T18:45:35.435540+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6027,"ondisk_log_size":6027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28770470,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6025,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":463481132,"num_objects":56592,"num_object_clones":0,"num_object_copies":226368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56592,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98133,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10472,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":923181056,"data_stored":923181056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98160,"ondisk_log_size":98160,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215179,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416380,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":189695,"kb_avail":93955460,"statfs":{"total":96636764160,"available":96210391040,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194247732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.751}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.57399999999999995},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412476,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":185792,"kb_avail":93959364,"statfs":{"total":96636764160,"available":96214388736,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":190251008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.62},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.75900000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420540,"kb_used_data":226676,"kb_used_omap":0,"kb_used_meta":193855,"kb_avail":93951300,"statfs":{"total":96636764160,"available":96206131200,"internally_reserved":0,"allocated":232116224,"data_stored":231892407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198507572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":0.66400000000000003},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.75800000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397440,"kb_used_data":222968,"kb_used_omap":0,"kb_used_meta":174463,"kb_avail":93974400,"statfs":{"total":96636764160,"available":96229785600,"internally_reserved":0,"allocated":228319232,"data_stored":228095415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":178650164},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,41],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.871},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.67400000000000004}]}]}]}} 2021-01-15T18:45:50.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:52.339 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:52.368 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:52.390 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2021-01-15T18:45:50.551806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":454969644,"num_objects":55553,"num_object_clones":0,"num_object_copies":222212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99172,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10567,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99199,"ondisk_log_size":99199,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1657860,"kb_used_data":890376,"kb_used_omap":2,"kb_used_meta":767293,"kb_avail":375829500,"statfs":{"total":386547056640,"available":384849408000,"internally_reserved":0,"allocated":911745024,"data_stored":910849756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":785708188},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,6,13,19,9,0,0,5,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":102,"apply_latency_ms":102,"commit_latency_ns":102000000,"apply_latency_ns":102000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9789440,"num_objects":-1195,"num_object_clones":0,"num_object_copies":-4780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1256,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"1.f","version":"42'6049","reported_seq":"6304","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.764003+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:49.764003+0000","last_peered":"2021-01-15T18:45:49.764003+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:49.764003+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6049,"ondisk_log_size":6049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28336128,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6098","reported_seq":"9024","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.960163+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:49.960163+0000","last_peered":"2021-01-15T18:45:49.960163+0000","last_clean":"2021-01-15T18:45:49.960163+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:49.960163+0000","last_undegraded":"2021-01-15T18:45:49.960163+0000","last_fullsized":"2021-01-15T18:45:49.960163+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6098,"ondisk_log_size":6098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27549862,"num_objects":3364,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6111","reported_seq":"6259","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:48.163948+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:48.163948+0000","last_peered":"2021-01-15T18:45:48.163948+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:48.163948+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6111,"ondisk_log_size":6111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27693163,"num_objects":3382,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6110,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6283","reported_seq":"6525","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.842619+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:49.842619+0000","last_peered":"2021-01-15T18:45:49.842619+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:49.842619+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6283,"ondisk_log_size":6283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28745728,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6368","reported_seq":"6962","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.912922+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:49.912922+0000","last_peered":"2021-01-15T18:45:49.912922+0000","last_clean":"2021-01-15T18:45:49.912922+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:49.912922+0000","last_undegraded":"2021-01-15T18:45:49.912922+0000","last_fullsized":"2021-01-15T18:45:49.912922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6368,"ondisk_log_size":6368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28344486,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6060","reported_seq":"6273","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.834950+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:49.834950+0000","last_peered":"2021-01-15T18:45:49.834950+0000","last_clean":"2021-01-15T18:45:49.834950+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:49.834950+0000","last_undegraded":"2021-01-15T18:45:49.834950+0000","last_fullsized":"2021-01-15T18:45:49.834950+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6060,"ondisk_log_size":6060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28393638,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6058,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6201","reported_seq":"6451","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.916695+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:49.916695+0000","last_peered":"2021-01-15T18:45:49.916695+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:49.916695+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6201,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28868774,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6318","reported_seq":"6474","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:48.437350+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:48.437350+0000","last_peered":"2021-01-15T18:45:48.437350+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:48.437350+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6318,"ondisk_log_size":6318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28987475,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6317,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6258","reported_seq":"9071","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.917476+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:49.917476+0000","last_peered":"2021-01-15T18:45:49.917476+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:49.917476+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6258,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28430419,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6260","reported_seq":"9203","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.902943+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:49.902943+0000","last_peered":"2021-01-15T18:45:49.902943+0000","last_clean":"2021-01-15T18:45:49.902943+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:49.902943+0000","last_undegraded":"2021-01-15T18:45:49.902943+0000","last_fullsized":"2021-01-15T18:45:49.902943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6260,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28651769,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6118","reported_seq":"6354","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.844185+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:49.844185+0000","last_peered":"2021-01-15T18:45:49.844185+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:49.844185+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28147878,"num_objects":3437,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"42'6118","reported_seq":"8949","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.649640+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:49.649640+0000","last_peered":"2021-01-15T18:45:49.649640+0000","last_clean":"2021-01-15T18:45:49.649640+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:49.649640+0000","last_undegraded":"2021-01-15T18:45:49.649640+0000","last_fullsized":"2021-01-15T18:45:49.649640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28254374,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6085","reported_seq":"8980","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.396613+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:49.396613+0000","last_peered":"2021-01-15T18:45:49.396613+0000","last_clean":"2021-01-15T18:45:49.396613+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:49.396613+0000","last_undegraded":"2021-01-15T18:45:49.396613+0000","last_fullsized":"2021-01-15T18:45:49.396613+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6085,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27771046,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6340","reported_seq":"6478","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.383345+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:49.383345+0000","last_peered":"2021-01-15T18:45:49.383345+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:49.383345+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6340,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29048998,"num_objects":3547,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6451","reported_seq":"6711","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.867024+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:49.867024+0000","last_peered":"2021-01-15T18:45:49.867024+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:49.867024+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6451,"ondisk_log_size":6451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29417804,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6081","reported_seq":"8948","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:48.501576+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:48.501576+0000","last_peered":"2021-01-15T18:45:48.501576+0000","last_clean":"2021-01-15T18:45:48.501576+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:48.501576+0000","last_undegraded":"2021-01-15T18:45:48.501576+0000","last_fullsized":"2021-01-15T18:45:48.501576+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6081,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28328102,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":454969644,"num_objects":55553,"num_object_clones":0,"num_object_copies":222212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99172,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10567,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909041664,"data_stored":909041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99199,"ondisk_log_size":99199,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418552,"kb_used_data":222808,"kb_used_omap":0,"kb_used_meta":195711,"kb_avail":93953288,"statfs":{"total":96636764160,"available":96208166912,"internally_reserved":0,"allocated":228155392,"data_stored":227931575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":200408116},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,9,1,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.71899999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.81299999999999994}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":2.8100000000000001},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.286}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419136,"kb_used_data":223564,"kb_used_omap":0,"kb_used_meta":195520,"kb_avail":93952704,"statfs":{"total":96636764160,"available":96207568896,"internally_reserved":0,"allocated":228929536,"data_stored":228705719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":200212480},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,7,6,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":1.3819999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.595},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":1.2450000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426880,"kb_used_data":222920,"kb_used_omap":0,"kb_used_meta":203903,"kb_avail":93944960,"statfs":{"total":96636764160,"available":96199639040,"internally_reserved":0,"allocated":228270080,"data_stored":228046263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208796724},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":11.929},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.4160000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":9.7759999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":9.9979999999999993}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":10.237},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":10.326000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393292,"kb_used_data":221084,"kb_used_omap":0,"kb_used_meta":172159,"kb_avail":93978548,"statfs":{"total":96636764160,"available":96234033152,"internally_reserved":0,"allocated":226390016,"data_stored":226166199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":176290868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.7549999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.5169999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.6200000000000001}]}]}]}} 2021-01-15T18:45:52.392 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:53.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:53.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10840/222212 objects degraded (4.878%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":403,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55553,"data_bytes":454969644,"bytes_used":1697648640,"bytes_avail":384849408000,"bytes_total":386547056640,"degraded_objects":10840,"degraded_total":222212,"degraded_ratio":0.048782243983223228,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":742,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":88},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:53.464 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:53.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:54.268 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:54.298 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:54.324 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2021-01-15T18:45:53.644915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":453544236,"num_objects":55379,"num_object_clones":0,"num_object_copies":221516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99346,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10567,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99373,"ondisk_log_size":99373,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1660156,"kb_used_data":886952,"kb_used_omap":2,"kb_used_meta":773053,"kb_avail":375827204,"statfs":{"total":386547056640,"available":384847056896,"internally_reserved":0,"allocated":908238848,"data_stored":907343580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":791606428},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,4,13,12,3,0,2,5,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10084352,"num_objects":-1231,"num_object_clones":0,"num_object_copies":-4924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.094972"},"pg_stats":[{"pgid":"1.f","version":"42'6049","reported_seq":"6304","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.764003+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:49.764003+0000","last_peered":"2021-01-15T18:45:49.764003+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:49.764003+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6049,"ondisk_log_size":6049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28336128,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"42'6098","reported_seq":"9024","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.960163+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:49.960163+0000","last_peered":"2021-01-15T18:45:49.960163+0000","last_clean":"2021-01-15T18:45:49.960163+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:49.960163+0000","last_undegraded":"2021-01-15T18:45:49.960163+0000","last_fullsized":"2021-01-15T18:45:49.960163+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6098,"ondisk_log_size":6098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27549862,"num_objects":3364,"num_object_clones":0,"num_object_copies":13456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6161","reported_seq":"6309","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.523574+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:53.523574+0000","last_peered":"2021-01-15T18:45:53.523574+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:53.523574+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6161,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27283563,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6160,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1947}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"42'6283","reported_seq":"6525","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.842619+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:49.842619+0000","last_peered":"2021-01-15T18:45:49.842619+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:49.842619+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6283,"ondisk_log_size":6283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28745728,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"42'6368","reported_seq":"6962","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.912922+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:49.912922+0000","last_peered":"2021-01-15T18:45:49.912922+0000","last_clean":"2021-01-15T18:45:49.912922+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:49.912922+0000","last_undegraded":"2021-01-15T18:45:49.912922+0000","last_fullsized":"2021-01-15T18:45:49.912922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6368,"ondisk_log_size":6368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28344486,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"42'6060","reported_seq":"6273","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.834950+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:49.834950+0000","last_peered":"2021-01-15T18:45:49.834950+0000","last_clean":"2021-01-15T18:45:49.834950+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:49.834950+0000","last_undegraded":"2021-01-15T18:45:49.834950+0000","last_fullsized":"2021-01-15T18:45:49.834950+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6060,"ondisk_log_size":6060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28393638,"num_objects":3467,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6058,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"42'6201","reported_seq":"6451","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.916695+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:49.916695+0000","last_peered":"2021-01-15T18:45:49.916695+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:49.916695+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6201,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28868774,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6376","reported_seq":"6532","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.574950+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:53.574950+0000","last_peered":"2021-01-15T18:45:53.574950+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:53.574950+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6376,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28512339,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6258","reported_seq":"9071","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.917476+0000","last_change":"2021-01-15T18:45:33.281426+0000","last_active":"2021-01-15T18:45:49.917476+0000","last_peered":"2021-01-15T18:45:49.917476+0000","last_clean":"2021-01-15T18:39:28.601024+0000","last_became_active":"2021-01-15T18:40:32.747391+0000","last_became_peered":"2021-01-15T18:40:32.747391+0000","last_unstale":"2021-01-15T18:45:49.917476+0000","last_undegraded":"2021-01-15T18:40:32.524535+0000","last_fullsized":"2021-01-15T18:40:32.483440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6258,"ondisk_log_size":6258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28430419,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3416}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"42'6260","reported_seq":"9203","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.902943+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:49.902943+0000","last_peered":"2021-01-15T18:45:49.902943+0000","last_clean":"2021-01-15T18:45:49.902943+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:49.902943+0000","last_undegraded":"2021-01-15T18:45:49.902943+0000","last_fullsized":"2021-01-15T18:45:49.902943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6260,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28651769,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"42'6118","reported_seq":"6354","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.844185+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:49.844185+0000","last_peered":"2021-01-15T18:45:49.844185+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:49.844185+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28147878,"num_objects":3437,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"42'6118","reported_seq":"8949","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.649640+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:49.649640+0000","last_peered":"2021-01-15T18:45:49.649640+0000","last_clean":"2021-01-15T18:45:49.649640+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:49.649640+0000","last_undegraded":"2021-01-15T18:45:49.649640+0000","last_fullsized":"2021-01-15T18:45:49.649640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28254374,"num_objects":3450,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'6085","reported_seq":"8980","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:49.396613+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:49.396613+0000","last_peered":"2021-01-15T18:45:49.396613+0000","last_clean":"2021-01-15T18:45:49.396613+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:49.396613+0000","last_undegraded":"2021-01-15T18:45:49.396613+0000","last_fullsized":"2021-01-15T18:45:49.396613+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6085,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27771046,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"42'6340","reported_seq":"6478","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.383345+0000","last_change":"2021-01-15T18:40:32.789793+0000","last_active":"2021-01-15T18:45:49.383345+0000","last_peered":"2021-01-15T18:45:49.383345+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:49.383345+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6340,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29048998,"num_objects":3547,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2139}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"42'6451","reported_seq":"6711","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:49.867024+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:49.867024+0000","last_peered":"2021-01-15T18:45:49.867024+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:49.867024+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6451,"ondisk_log_size":6451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29417804,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6147","reported_seq":"9014","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:53.571619+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:53.571619+0000","last_peered":"2021-01-15T18:45:53.571619+0000","last_clean":"2021-01-15T18:45:53.571619+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:53.571619+0000","last_undegraded":"2021-01-15T18:45:53.571619+0000","last_fullsized":"2021-01-15T18:45:53.571619+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6147,"ondisk_log_size":6147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27787430,"num_objects":3393,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":453544236,"num_objects":55379,"num_object_clones":0,"num_object_copies":221516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99346,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10567,"num_bytes_recovered":68706304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":904761344,"data_stored":904761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99373,"ondisk_log_size":99373,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215180,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418552,"kb_used_data":222808,"kb_used_omap":0,"kb_used_meta":195711,"kb_avail":93953288,"statfs":{"total":96636764160,"available":96208166912,"internally_reserved":0,"allocated":228155392,"data_stored":227931575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":200408116},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,9,1,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.71899999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.81299999999999994}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":2.8100000000000001},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.286}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421432,"kb_used_data":220140,"kb_used_omap":0,"kb_used_meta":201280,"kb_avail":93950408,"statfs":{"total":96636764160,"available":96205217792,"internally_reserved":0,"allocated":225423360,"data_stored":225199543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":206110720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.751}]}]},{"osd":0,"up_from":22,"seq":94489280592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426880,"kb_used_data":222920,"kb_used_omap":0,"kb_used_meta":203903,"kb_avail":93944960,"statfs":{"total":96636764160,"available":96199639040,"internally_reserved":0,"allocated":228270080,"data_stored":228046263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208796724},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":11.929},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.4160000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:44:49 2021","interfaces":[{"interface":"back","average":{"1min":4.0099999999999998,"5min":3.2240000000000002,"15min":2.5539999999999998},"min":{"1min":0.51200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":13.337999999999999,"5min":27.77,"15min":27.77},"last":9.7759999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":3.2029999999999998,"15min":2.6150000000000002},"min":{"1min":0.47499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.3019999999999996,"5min":22.765000000000001,"15min":22.765000000000001},"last":9.9979999999999993}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":10.237},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":10.326000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393292,"kb_used_data":221084,"kb_used_omap":0,"kb_used_meta":172159,"kb_avail":93978548,"statfs":{"total":96636764160,"available":96234033152,"internally_reserved":0,"allocated":226390016,"data_stored":226166199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":176290868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.7549999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.5169999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.6200000000000001}]}]}]}} 2021-01-15T18:45:57.327 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:57.632 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:57.634 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:57.646 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2021-01-15T18:45:56.553221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447260972,"num_objects":54612,"num_object_clones":0,"num_object_copies":218448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100113,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10664,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100140,"ondisk_log_size":100140,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1643460,"kb_used_data":876592,"kb_used_omap":2,"kb_used_meta":766781,"kb_avail":375843900,"statfs":{"total":386547056640,"available":384864153600,"internally_reserved":0,"allocated":897630208,"data_stored":896720402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":785183900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,5,12,6,5,4,1,4,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":101,"apply_latency_ms":101,"commit_latency_ns":101000000,"apply_latency_ns":101000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16220160,"num_objects":-1980,"num_object_clones":0,"num_object_copies":-7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1980,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001872"},"pg_stats":[{"pgid":"1.f","version":"44'6118","reported_seq":"6380","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.659116+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.659116+0000","last_peered":"2021-01-15T18:45:54.659116+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.659116+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27770880,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6161","reported_seq":"6309","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.523574+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:53.523574+0000","last_peered":"2021-01-15T18:45:53.523574+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:53.523574+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6161,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27283563,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6160,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1947}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"43'6337","reported_seq":"6587","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.561270+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.561270+0000","last_peered":"2021-01-15T18:45:54.561270+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.561270+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6337,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28303360,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"43'6426","reported_seq":"7022","reported_epoch":"43","state":"active+clean","last_fresh":"2021-01-15T18:45:54.524534+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:54.524534+0000","last_peered":"2021-01-15T18:45:54.524534+0000","last_clean":"2021-01-15T18:45:54.524534+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:54.524534+0000","last_undegraded":"2021-01-15T18:45:54.524534+0000","last_fullsized":"2021-01-15T18:45:54.524534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6426,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"43'6263","reported_seq":"6523","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.501884+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.501884+0000","last_peered":"2021-01-15T18:45:54.501884+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.501884+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6263,"ondisk_log_size":6263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28360870,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6376","reported_seq":"6532","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.574950+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:53.574950+0000","last_peered":"2021-01-15T18:45:53.574950+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:53.574950+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6376,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28512339,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6323","reported_seq":"9268","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.658767+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.658767+0000","last_peered":"2021-01-15T18:45:54.658767+0000","last_clean":"2021-01-15T18:45:54.658767+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.658767+0000","last_undegraded":"2021-01-15T18:45:54.658767+0000","last_fullsized":"2021-01-15T18:45:54.658767+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6323,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28135673,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"43'6165","reported_seq":"6409","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.622808+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:54.622808+0000","last_peered":"2021-01-15T18:45:54.622808+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:54.622808+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6165,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762854,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"43'6150","reported_seq":"9048","reported_epoch":"43","state":"active+clean","last_fresh":"2021-01-15T18:45:54.244077+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:54.244077+0000","last_peered":"2021-01-15T18:45:54.244077+0000","last_clean":"2021-01-15T18:45:54.244077+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:54.244077+0000","last_undegraded":"2021-01-15T18:45:54.244077+0000","last_fullsized":"2021-01-15T18:45:54.244077+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6150,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27238566,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6395","reported_seq":"6598","reported_epoch":"43","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.413012+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:45:54.413012+0000","last_peered":"2021-01-15T18:45:54.413012+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:54.413012+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6395,"ondisk_log_size":6395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28598438,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6526","reported_seq":"6791","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.795911+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:54.795911+0000","last_peered":"2021-01-15T18:45:54.795911+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:54.795911+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6526,"ondisk_log_size":6526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28803404,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6523,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6147","reported_seq":"9014","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:53.571619+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:53.571619+0000","last_peered":"2021-01-15T18:45:53.571619+0000","last_clean":"2021-01-15T18:45:53.571619+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:53.571619+0000","last_undegraded":"2021-01-15T18:45:53.571619+0000","last_fullsized":"2021-01-15T18:45:53.571619+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6147,"ondisk_log_size":6147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27787430,"num_objects":3393,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447260972,"num_objects":54612,"num_object_clones":0,"num_object_copies":218448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100113,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10664,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":894607360,"data_stored":894607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100140,"ondisk_log_size":100140,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408280,"kb_used_data":218696,"kb_used_omap":0,"kb_used_meta":189567,"kb_avail":93963560,"statfs":{"total":96636764160,"available":96218685440,"internally_reserved":0,"allocated":223944704,"data_stored":223716041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194116660},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":9.8740000000000006},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.036}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.3879999999999999},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.492}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":5.3920000000000003},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.1419999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421432,"kb_used_data":220140,"kb_used_omap":0,"kb_used_meta":201280,"kb_avail":93950408,"statfs":{"total":96636764160,"available":96205217792,"internally_reserved":0,"allocated":225423360,"data_stored":225199543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":206110720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.751}]}]},{"osd":0,"up_from":22,"seq":94489280593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412504,"kb_used_data":218824,"kb_used_omap":0,"kb_used_meta":193663,"kb_avail":93959336,"statfs":{"total":96636764160,"available":96214360064,"internally_reserved":0,"allocated":224075776,"data_stored":223847113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198310964},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.624}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.81299999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401244,"kb_used_data":218932,"kb_used_omap":0,"kb_used_meta":182271,"kb_avail":93970596,"statfs":{"total":96636764160,"available":96225890304,"internally_reserved":0,"allocated":224186368,"data_stored":223957705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":186645556},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,6,3,5,2,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":4.1210000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.2889999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":4.0350000000000001},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":4.0819999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":4.5999999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":4.4969999999999999}]}]}]}} 2021-01-15T18:45:57.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:45:58.016 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:58.017 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10743/218448 objects degraded (4.918%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":408,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54612,"data_bytes":447260972,"bytes_used":1682903040,"bytes_avail":384864153600,"bytes_total":386547056640,"degraded_objects":10743,"degraded_total":218448,"degraded_ratio":0.049178751922654362,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":12285,"recovering_keys_per_sec":0,"num_objects_recovered":192,"num_bytes_recovered":98304,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":247},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:45:58.030 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:45:58.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:45:58.324 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:45:58.325 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:45:58.337 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2021-01-15T18:45:56.553221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":447260972,"num_objects":54612,"num_object_clones":0,"num_object_copies":218448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100113,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10664,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100140,"ondisk_log_size":100140,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1643460,"kb_used_data":876592,"kb_used_omap":2,"kb_used_meta":766781,"kb_avail":375843900,"statfs":{"total":386547056640,"available":384864153600,"internally_reserved":0,"allocated":897630208,"data_stored":896720402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":785183900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,5,12,6,5,4,1,4,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":101,"apply_latency_ms":101,"commit_latency_ns":101000000,"apply_latency_ns":101000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16220160,"num_objects":-1980,"num_object_clones":0,"num_object_copies":-7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1980,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001872"},"pg_stats":[{"pgid":"1.f","version":"44'6118","reported_seq":"6380","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.659116+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.659116+0000","last_peered":"2021-01-15T18:45:54.659116+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.659116+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27770880,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2140}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"42'6161","reported_seq":"6309","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.523574+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:53.523574+0000","last_peered":"2021-01-15T18:45:53.523574+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:53.523574+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6161,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27283563,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6160,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1947}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"43'6337","reported_seq":"6587","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.561270+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.561270+0000","last_peered":"2021-01-15T18:45:54.561270+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.561270+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6337,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28303360,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"43'6426","reported_seq":"7022","reported_epoch":"43","state":"active+clean","last_fresh":"2021-01-15T18:45:54.524534+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:54.524534+0000","last_peered":"2021-01-15T18:45:54.524534+0000","last_clean":"2021-01-15T18:45:54.524534+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:54.524534+0000","last_undegraded":"2021-01-15T18:45:54.524534+0000","last_fullsized":"2021-01-15T18:45:54.524534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6426,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"43'6263","reported_seq":"6523","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.501884+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.501884+0000","last_peered":"2021-01-15T18:45:54.501884+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.501884+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6263,"ondisk_log_size":6263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28360870,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2170}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"42'6376","reported_seq":"6532","reported_epoch":"42","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:53.574950+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:53.574950+0000","last_peered":"2021-01-15T18:45:53.574950+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:53.574950+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6376,"ondisk_log_size":6376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28512339,"num_objects":3481,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6323","reported_seq":"9268","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.658767+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.658767+0000","last_peered":"2021-01-15T18:45:54.658767+0000","last_clean":"2021-01-15T18:45:54.658767+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.658767+0000","last_undegraded":"2021-01-15T18:45:54.658767+0000","last_fullsized":"2021-01-15T18:45:54.658767+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6323,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28135673,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"43'6165","reported_seq":"6409","reported_epoch":"43","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.622808+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:54.622808+0000","last_peered":"2021-01-15T18:45:54.622808+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:54.622808+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6165,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762854,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"43'6150","reported_seq":"9048","reported_epoch":"43","state":"active+clean","last_fresh":"2021-01-15T18:45:54.244077+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:54.244077+0000","last_peered":"2021-01-15T18:45:54.244077+0000","last_clean":"2021-01-15T18:45:54.244077+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:54.244077+0000","last_undegraded":"2021-01-15T18:45:54.244077+0000","last_fullsized":"2021-01-15T18:45:54.244077+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6150,"ondisk_log_size":6150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27238566,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6395","reported_seq":"6598","reported_epoch":"43","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.413012+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:45:54.413012+0000","last_peered":"2021-01-15T18:45:54.413012+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:54.413012+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6395,"ondisk_log_size":6395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28598438,"num_objects":3492,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6526","reported_seq":"6791","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.795911+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:54.795911+0000","last_peered":"2021-01-15T18:45:54.795911+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:54.795911+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6526,"ondisk_log_size":6526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28803404,"num_objects":3518,"num_object_clones":0,"num_object_copies":14072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6523,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"42'6147","reported_seq":"9014","reported_epoch":"42","state":"active+clean","last_fresh":"2021-01-15T18:45:53.571619+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:53.571619+0000","last_peered":"2021-01-15T18:45:53.571619+0000","last_clean":"2021-01-15T18:45:53.571619+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:53.571619+0000","last_undegraded":"2021-01-15T18:45:53.571619+0000","last_fullsized":"2021-01-15T18:45:53.571619+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6147,"ondisk_log_size":6147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27787430,"num_objects":3393,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":447260972,"num_objects":54612,"num_object_clones":0,"num_object_copies":218448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100113,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10664,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":894607360,"data_stored":894607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100140,"ondisk_log_size":100140,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408280,"kb_used_data":218696,"kb_used_omap":0,"kb_used_meta":189567,"kb_avail":93963560,"statfs":{"total":96636764160,"available":96218685440,"internally_reserved":0,"allocated":223944704,"data_stored":223716041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194116660},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,6,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":9.8740000000000006},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.036}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.3879999999999999},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.492}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":5.3920000000000003},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":1.1419999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421432,"kb_used_data":220140,"kb_used_omap":0,"kb_used_meta":201280,"kb_avail":93950408,"statfs":{"total":96636764160,"available":96205217792,"internally_reserved":0,"allocated":225423360,"data_stored":225199543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":206110720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:44:51 2021","interfaces":[{"interface":"back","average":{"1min":5.1180000000000003,"5min":3.8199999999999998,"15min":3.7759999999999998},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.468,"5min":36.380000000000003,"15min":36.380000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":4.133,"5min":2.7519999999999998,"15min":1.5740000000000001},"min":{"1min":0.62,"5min":0.371,"15min":0.371},"max":{"1min":23.98,"5min":23.98,"15min":23.98},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:44:55 2021","interfaces":[{"interface":"back","average":{"1min":4.2709999999999999,"5min":3.0609999999999999,"15min":2.5470000000000002},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":21.952000000000002,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.7730000000000001,"5min":3.1400000000000001,"15min":2.5609999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":18.34,"5min":18.34,"15min":18.34},"last":0.751}]}]},{"osd":0,"up_from":22,"seq":94489280593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412504,"kb_used_data":218824,"kb_used_omap":0,"kb_used_meta":193663,"kb_avail":93959336,"statfs":{"total":96636764160,"available":96214360064,"internally_reserved":0,"allocated":224075776,"data_stored":223847113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198310964},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.624}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:44:57 2021","interfaces":[{"interface":"back","average":{"1min":4.5529999999999999,"5min":3.5350000000000001,"15min":4.3070000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":12.702999999999999,"5min":41.984999999999999,"15min":41.984999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.7810000000000001,"5min":4.6799999999999997,"15min":2.98},"min":{"1min":0.61699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":12.805999999999999,"5min":56.927,"15min":56.927},"last":0.81299999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401244,"kb_used_data":218932,"kb_used_omap":0,"kb_used_meta":182271,"kb_avail":93970596,"statfs":{"total":96636764160,"available":96225890304,"internally_reserved":0,"allocated":224186368,"data_stored":223957705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":186645556},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,6,3,5,2,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":4.1210000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":4.2889999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":4.0350000000000001},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":4.0819999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":4.5999999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":4.4969999999999999}]}]}]}} 2021-01-15T18:46:01.340 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:01.642 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:01.656 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2021-01-15T18:46:00.553971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11196,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1638700,"kb_used_data":873740,"kb_used_omap":2,"kb_used_meta":764925,"kb_avail":375848660,"statfs":{"total":386547056640,"available":384869027840,"internally_reserved":0,"allocated":894709760,"data_stored":893775640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":783283356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8273920,"num_objects":-1010,"num_object_clones":0,"num_object_copies":-4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001642"},"pg_stats":[{"pgid":"1.f","version":"44'6119","reported_seq":"6381","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.729201+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.729201+0000","last_peered":"2021-01-15T18:45:54.729201+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.729201+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6119,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"6336","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.067112+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:55.067112+0000","last_peered":"2021-01-15T18:45:55.067112+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:55.067112+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"44'6338","reported_seq":"6588","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.658868+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.658868+0000","last_peered":"2021-01-15T18:45:54.658868+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.658868+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6338,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"44'6428","reported_seq":"7024","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.077322+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:55.077322+0000","last_peered":"2021-01-15T18:45:55.077322+0000","last_clean":"2021-01-15T18:45:55.077322+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:55.077322+0000","last_undegraded":"2021-01-15T18:45:55.077322+0000","last_fullsized":"2021-01-15T18:45:55.077322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27852966,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'6264","reported_seq":"6524","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.657750+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.657750+0000","last_peered":"2021-01-15T18:45:54.657750+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.657750+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6264,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28352678,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6554","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.164465+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:55.164465+0000","last_peered":"2021-01-15T18:45:55.164465+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:55.164465+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6324","reported_seq":"9269","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.707740+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.707740+0000","last_peered":"2021-01-15T18:45:54.707740+0000","last_clean":"2021-01-15T18:45:54.707740+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.707740+0000","last_undegraded":"2021-01-15T18:45:54.707740+0000","last_fullsized":"2021-01-15T18:45:54.707740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6324,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28127481,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"44'6167","reported_seq":"6411","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.137537+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:55.137537+0000","last_peered":"2021-01-15T18:45:55.137537+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:55.137537+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6167,"ondisk_log_size":6167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27746470,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'6156","reported_seq":"9055","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.179701+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:55.179701+0000","last_peered":"2021-01-15T18:45:55.179701+0000","last_clean":"2021-01-15T18:45:55.179701+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:55.179701+0000","last_undegraded":"2021-01-15T18:45:55.179701+0000","last_fullsized":"2021-01-15T18:45:55.179701+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6156,"ondisk_log_size":6156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27189414,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"7656","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:59.428062+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:45:59.428062+0000","last_peered":"2021-01-15T18:45:59.428062+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:59.428062+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6527","reported_seq":"6793","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.058876+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:55.058876+0000","last_peered":"2021-01-15T18:45:55.058876+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:55.058876+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6527,"ondisk_log_size":6527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28795212,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9032","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.207885+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:55.207885+0000","last_peered":"2021-01-15T18:45:55.207885+0000","last_clean":"2021-01-15T18:45:55.207885+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:55.207885+0000","last_undegraded":"2021-01-15T18:45:55.207885+0000","last_fullsized":"2021-01-15T18:45:55.207885+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11196,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893235200,"data_stored":893235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412224,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193791,"kb_avail":93959616,"statfs":{"total":96636764160,"available":96214646784,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198442036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.496}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.755},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412416,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959424,"statfs":{"total":96636764160,"available":96214450176,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.628}]}]},{"osd":0,"up_from":22,"seq":94489280594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416448,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":198015,"kb_avail":93955392,"statfs":{"total":96636764160,"available":96210321408,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202767412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.47799999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.58999999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397612,"kb_used_data":218468,"kb_used_omap":0,"kb_used_meta":179135,"kb_avail":93974228,"statfs":{"total":96636764160,"available":96229609472,"internally_reserved":0,"allocated":223711232,"data_stored":223477702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183434292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.76000000000000001}]}]}]}} 2021-01-15T18:46:01.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:02.002 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:02.002 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10211/218172 objects degraded (4.680%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54543,"data_bytes":446695724,"bytes_used":1678028800,"bytes_avail":384869027840,"bytes_total":386547056640,"degraded_objects":10211,"degraded_total":218172,"degraded_ratio":0.046802522780191778,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":12285,"recovering_keys_per_sec":0,"num_objects_recovered":629,"num_bytes_recovered":98304,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":126},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:02.015 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:02.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:02.337 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:02.339 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:02.352 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2021-01-15T18:46:00.553971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11196,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1638700,"kb_used_data":873740,"kb_used_omap":2,"kb_used_meta":764925,"kb_avail":375848660,"statfs":{"total":386547056640,"available":384869027840,"internally_reserved":0,"allocated":894709760,"data_stored":893775640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":783283356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8273920,"num_objects":-1010,"num_object_clones":0,"num_object_copies":-4040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001642"},"pg_stats":[{"pgid":"1.f","version":"44'6119","reported_seq":"6381","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.729201+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.729201+0000","last_peered":"2021-01-15T18:45:54.729201+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.729201+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6119,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"6336","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.067112+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:55.067112+0000","last_peered":"2021-01-15T18:45:55.067112+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:55.067112+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"44'6338","reported_seq":"6588","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.658868+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.658868+0000","last_peered":"2021-01-15T18:45:54.658868+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.658868+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6338,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"44'6428","reported_seq":"7024","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.077322+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:55.077322+0000","last_peered":"2021-01-15T18:45:55.077322+0000","last_clean":"2021-01-15T18:45:55.077322+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:55.077322+0000","last_undegraded":"2021-01-15T18:45:55.077322+0000","last_fullsized":"2021-01-15T18:45:55.077322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27852966,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'6264","reported_seq":"6524","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.657750+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.657750+0000","last_peered":"2021-01-15T18:45:54.657750+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.657750+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6264,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28352678,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6554","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.164465+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:55.164465+0000","last_peered":"2021-01-15T18:45:55.164465+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:55.164465+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6324","reported_seq":"9269","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.707740+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.707740+0000","last_peered":"2021-01-15T18:45:54.707740+0000","last_clean":"2021-01-15T18:45:54.707740+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.707740+0000","last_undegraded":"2021-01-15T18:45:54.707740+0000","last_fullsized":"2021-01-15T18:45:54.707740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6324,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28127481,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"44'6167","reported_seq":"6411","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.137537+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:55.137537+0000","last_peered":"2021-01-15T18:45:55.137537+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:55.137537+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6167,"ondisk_log_size":6167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27746470,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'6156","reported_seq":"9055","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.179701+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:55.179701+0000","last_peered":"2021-01-15T18:45:55.179701+0000","last_clean":"2021-01-15T18:45:55.179701+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:55.179701+0000","last_undegraded":"2021-01-15T18:45:55.179701+0000","last_fullsized":"2021-01-15T18:45:55.179701+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6156,"ondisk_log_size":6156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27189414,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"7656","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:59.428062+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:45:59.428062+0000","last_peered":"2021-01-15T18:45:59.428062+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:45:59.428062+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2640}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6527","reported_seq":"6793","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.058876+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:55.058876+0000","last_peered":"2021-01-15T18:45:55.058876+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:55.058876+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6527,"ondisk_log_size":6527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28795212,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9032","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.207885+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:55.207885+0000","last_peered":"2021-01-15T18:45:55.207885+0000","last_clean":"2021-01-15T18:45:55.207885+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:55.207885+0000","last_undegraded":"2021-01-15T18:45:55.207885+0000","last_fullsized":"2021-01-15T18:45:55.207885+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11196,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893235200,"data_stored":893235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412224,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193791,"kb_avail":93959616,"statfs":{"total":96636764160,"available":96214646784,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198442036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.496}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.755},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412416,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959424,"statfs":{"total":96636764160,"available":96214450176,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.628}]}]},{"osd":0,"up_from":22,"seq":94489280594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416448,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":198015,"kb_avail":93955392,"statfs":{"total":96636764160,"available":96210321408,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202767412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.47799999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.58999999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397612,"kb_used_data":218468,"kb_used_omap":0,"kb_used_meta":179135,"kb_avail":93974228,"statfs":{"total":96636764160,"available":96229609472,"internally_reserved":0,"allocated":223711232,"data_stored":223477702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183434292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.76000000000000001}]}]}]}} 2021-01-15T18:46:05.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:05.633 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:05.634 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:05.646 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2021-01-15T18:46:04.554733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1638700,"kb_used_data":873740,"kb_used_omap":2,"kb_used_meta":764925,"kb_avail":375848660,"statfs":{"total":386547056640,"available":384869027840,"internally_reserved":0,"allocated":894709760,"data_stored":893775640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":783283356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,19,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5865472,"num_objects":-716,"num_object_clones":0,"num_object_copies":-2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001989"},"pg_stats":[{"pgid":"1.f","version":"44'6119","reported_seq":"6381","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.729201+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.729201+0000","last_peered":"2021-01-15T18:45:54.729201+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.729201+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6119,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"6336","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.067112+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:55.067112+0000","last_peered":"2021-01-15T18:45:55.067112+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:55.067112+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"44'6338","reported_seq":"6588","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.658868+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.658868+0000","last_peered":"2021-01-15T18:45:54.658868+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.658868+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6338,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"44'6428","reported_seq":"7024","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.077322+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:55.077322+0000","last_peered":"2021-01-15T18:45:55.077322+0000","last_clean":"2021-01-15T18:45:55.077322+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:55.077322+0000","last_undegraded":"2021-01-15T18:45:55.077322+0000","last_fullsized":"2021-01-15T18:45:55.077322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27852966,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'6264","reported_seq":"6524","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.657750+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.657750+0000","last_peered":"2021-01-15T18:45:54.657750+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.657750+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6264,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28352678,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6554","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.164465+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:55.164465+0000","last_peered":"2021-01-15T18:45:55.164465+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:55.164465+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6324","reported_seq":"9269","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.707740+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.707740+0000","last_peered":"2021-01-15T18:45:54.707740+0000","last_clean":"2021-01-15T18:45:54.707740+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.707740+0000","last_undegraded":"2021-01-15T18:45:54.707740+0000","last_fullsized":"2021-01-15T18:45:54.707740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6324,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28127481,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"44'6167","reported_seq":"6411","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.137537+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:55.137537+0000","last_peered":"2021-01-15T18:45:55.137537+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:55.137537+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6167,"ondisk_log_size":6167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27746470,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'6156","reported_seq":"9055","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.179701+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:55.179701+0000","last_peered":"2021-01-15T18:45:55.179701+0000","last_clean":"2021-01-15T18:45:55.179701+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:55.179701+0000","last_undegraded":"2021-01-15T18:45:55.179701+0000","last_fullsized":"2021-01-15T18:45:55.179701+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6156,"ondisk_log_size":6156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27189414,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"8940","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:04.430707+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:46:04.430707+0000","last_peered":"2021-01-15T18:46:04.430707+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:46:04.430707+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6527","reported_seq":"6793","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.058876+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:55.058876+0000","last_peered":"2021-01-15T18:45:55.058876+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:55.058876+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6527,"ondisk_log_size":6527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28795212,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9032","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.207885+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:55.207885+0000","last_peered":"2021-01-15T18:45:55.207885+0000","last_clean":"2021-01-15T18:45:55.207885+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:55.207885+0000","last_undegraded":"2021-01-15T18:45:55.207885+0000","last_fullsized":"2021-01-15T18:45:55.207885+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893235200,"data_stored":893235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412224,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193791,"kb_avail":93959616,"statfs":{"total":96636764160,"available":96214646784,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198442036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.496}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.755},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412416,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959424,"statfs":{"total":96636764160,"available":96214450176,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.433},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.65300000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.58599999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416448,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":198015,"kb_avail":93955392,"statfs":{"total":96636764160,"available":96210321408,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202767412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.435},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.57999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397612,"kb_used_data":218468,"kb_used_omap":0,"kb_used_meta":179135,"kb_avail":93974228,"statfs":{"total":96636764160,"available":96229609472,"internally_reserved":0,"allocated":223711232,"data_stored":223477702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183434292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.76000000000000001}]}]}]}} 2021-01-15T18:46:05.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:06.024 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:06.024 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9569/218172 objects degraded (4.386%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":416,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54543,"data_bytes":446695724,"bytes_used":1678028800,"bytes_avail":384869027840,"bytes_total":386547056640,"degraded_objects":9569,"degraded_total":218172,"degraded_ratio":0.043859890361732948,"recovering_objects_per_sec":123,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1240,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":71},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:06.037 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:06.038 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:06.321 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:06.332 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2021-01-15T18:46:04.554733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1638700,"kb_used_data":873740,"kb_used_omap":2,"kb_used_meta":764925,"kb_avail":375848660,"statfs":{"total":386547056640,"available":384869027840,"internally_reserved":0,"allocated":894709760,"data_stored":893775640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":783283356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,19,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5865472,"num_objects":-716,"num_object_clones":0,"num_object_copies":-2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001989"},"pg_stats":[{"pgid":"1.f","version":"44'6119","reported_seq":"6381","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.729201+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.729201+0000","last_peered":"2021-01-15T18:45:54.729201+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.729201+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6119,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"6336","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.067112+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:55.067112+0000","last_peered":"2021-01-15T18:45:55.067112+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:55.067112+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"44'6338","reported_seq":"6588","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.658868+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.658868+0000","last_peered":"2021-01-15T18:45:54.658868+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.658868+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6338,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"44'6428","reported_seq":"7024","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.077322+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:55.077322+0000","last_peered":"2021-01-15T18:45:55.077322+0000","last_clean":"2021-01-15T18:45:55.077322+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:55.077322+0000","last_undegraded":"2021-01-15T18:45:55.077322+0000","last_fullsized":"2021-01-15T18:45:55.077322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27852966,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'6264","reported_seq":"6524","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.657750+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.657750+0000","last_peered":"2021-01-15T18:45:54.657750+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.657750+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6264,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28352678,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6554","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.164465+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:55.164465+0000","last_peered":"2021-01-15T18:45:55.164465+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:55.164465+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6324","reported_seq":"9269","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.707740+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.707740+0000","last_peered":"2021-01-15T18:45:54.707740+0000","last_clean":"2021-01-15T18:45:54.707740+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.707740+0000","last_undegraded":"2021-01-15T18:45:54.707740+0000","last_fullsized":"2021-01-15T18:45:54.707740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6324,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28127481,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"44'6167","reported_seq":"6411","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.137537+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:55.137537+0000","last_peered":"2021-01-15T18:45:55.137537+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:55.137537+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6167,"ondisk_log_size":6167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27746470,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'6156","reported_seq":"9055","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.179701+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:55.179701+0000","last_peered":"2021-01-15T18:45:55.179701+0000","last_clean":"2021-01-15T18:45:55.179701+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:55.179701+0000","last_undegraded":"2021-01-15T18:45:55.179701+0000","last_fullsized":"2021-01-15T18:45:55.179701+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6156,"ondisk_log_size":6156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27189414,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"8940","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:04.430707+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:46:04.430707+0000","last_peered":"2021-01-15T18:46:04.430707+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:46:04.430707+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6527","reported_seq":"6793","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.058876+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:55.058876+0000","last_peered":"2021-01-15T18:45:55.058876+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:55.058876+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6527,"ondisk_log_size":6527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28795212,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9032","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.207885+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:55.207885+0000","last_peered":"2021-01-15T18:45:55.207885+0000","last_clean":"2021-01-15T18:45:55.207885+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:55.207885+0000","last_undegraded":"2021-01-15T18:45:55.207885+0000","last_fullsized":"2021-01-15T18:45:55.207885+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893235200,"data_stored":893235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215182,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412224,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193791,"kb_avail":93959616,"statfs":{"total":96636764160,"available":96214646784,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198442036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.496}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.755},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412416,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959424,"statfs":{"total":96636764160,"available":96214450176,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.433},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.65300000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.58599999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416448,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":198015,"kb_avail":93955392,"statfs":{"total":96636764160,"available":96210321408,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202767412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.435},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.57999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397612,"kb_used_data":218468,"kb_used_omap":0,"kb_used_meta":179135,"kb_avail":93974228,"statfs":{"total":96636764160,"available":96229609472,"internally_reserved":0,"allocated":223711232,"data_stored":223477702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":183434292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.76000000000000001}]}]}]}} 2021-01-15T18:46:09.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:10.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:10.143 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:10.177 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2021-01-15T18:46:08.555473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1635436,"kb_used_data":873740,"kb_used_omap":2,"kb_used_meta":761661,"kb_avail":375851924,"statfs":{"total":386547056640,"available":384872370176,"internally_reserved":0,"allocated":894709760,"data_stored":893775640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":779941020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2,18],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001501"},"pg_stats":[{"pgid":"1.f","version":"44'6119","reported_seq":"6381","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.729201+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:45:54.729201+0000","last_peered":"2021-01-15T18:45:54.729201+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:45:54.729201+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6119,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762688,"num_objects":3389,"num_object_clones":0,"num_object_copies":13556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2139}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"44'6163","reported_seq":"9091","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.746348+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:45:54.746348+0000","last_peered":"2021-01-15T18:45:54.746348+0000","last_clean":"2021-01-15T18:45:54.746348+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:45:54.746348+0000","last_undegraded":"2021-01-15T18:45:54.746348+0000","last_fullsized":"2021-01-15T18:45:54.746348+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27017382,"num_objects":3299,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"6336","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.067112+0000","last_change":"2021-01-15T18:40:32.772306+0000","last_active":"2021-01-15T18:45:55.067112+0000","last_peered":"2021-01-15T18:45:55.067112+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:45:55.067112+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"44'6338","reported_seq":"6588","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.658868+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:45:54.658868+0000","last_peered":"2021-01-15T18:45:54.658868+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:45:54.658868+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6338,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28295168,"num_objects":3454,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2120}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"44'6428","reported_seq":"7024","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.077322+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:45:55.077322+0000","last_peered":"2021-01-15T18:45:55.077322+0000","last_clean":"2021-01-15T18:45:55.077322+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:45:55.077322+0000","last_undegraded":"2021-01-15T18:45:55.077322+0000","last_fullsized":"2021-01-15T18:45:55.077322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27852966,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"44'6122","reported_seq":"6337","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.828597+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:45:54.828597+0000","last_peered":"2021-01-15T18:45:54.828597+0000","last_clean":"2021-01-15T18:45:54.828597+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:45:54.828597+0000","last_undegraded":"2021-01-15T18:45:54.828597+0000","last_fullsized":"2021-01-15T18:45:54.828597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6122,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27885734,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"44'6264","reported_seq":"6524","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:54.657750+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:45:54.657750+0000","last_peered":"2021-01-15T18:45:54.657750+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:45:54.657750+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6264,"ondisk_log_size":6264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28352678,"num_objects":3462,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2169}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6554","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.164465+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:45:55.164465+0000","last_peered":"2021-01-15T18:45:55.164465+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:45:55.164465+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"42'6299","reported_seq":"9237","reported_epoch":"44","state":"active+clean+wait","last_fresh":"2021-01-15T18:45:54.725011+0000","last_change":"2021-01-15T18:45:54.725011+0000","last_active":"2021-01-15T18:45:54.725011+0000","last_peered":"2021-01-15T18:45:54.725011+0000","last_clean":"2021-01-15T18:45:54.725011+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:45:54.725011+0000","last_undegraded":"2021-01-15T18:45:54.725011+0000","last_fullsized":"2021-01-15T18:45:54.725011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6299,"ondisk_log_size":6299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28094547,"num_objects":3430,"num_object_clones":0,"num_object_copies":13720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"44'6324","reported_seq":"9269","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.707740+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:45:54.707740+0000","last_peered":"2021-01-15T18:45:54.707740+0000","last_clean":"2021-01-15T18:45:54.707740+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:45:54.707740+0000","last_undegraded":"2021-01-15T18:45:54.707740+0000","last_fullsized":"2021-01-15T18:45:54.707740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6324,"ondisk_log_size":6324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28127481,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"44'6167","reported_seq":"6411","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.137537+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:45:55.137537+0000","last_peered":"2021-01-15T18:45:55.137537+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:45:55.137537+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6167,"ondisk_log_size":6167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27746470,"num_objects":3388,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2096}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'6169","reported_seq":"9002","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:54.769794+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:45:54.769794+0000","last_peered":"2021-01-15T18:45:54.769794+0000","last_clean":"2021-01-15T18:45:54.769794+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:45:54.769794+0000","last_undegraded":"2021-01-15T18:45:54.769794+0000","last_fullsized":"2021-01-15T18:45:54.769794+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6169,"ondisk_log_size":6169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27836582,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6167,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"44'6156","reported_seq":"9055","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.179701+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:45:55.179701+0000","last_peered":"2021-01-15T18:45:55.179701+0000","last_clean":"2021-01-15T18:45:55.179701+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:45:55.179701+0000","last_undegraded":"2021-01-15T18:45:55.179701+0000","last_fullsized":"2021-01-15T18:45:55.179701+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6156,"ondisk_log_size":6156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27189414,"num_objects":3320,"num_object_clones":0,"num_object_copies":13280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"8940","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:04.430707+0000","last_change":"2021-01-15T18:45:52.634641+0000","last_active":"2021-01-15T18:46:04.430707+0000","last_peered":"2021-01-15T18:46:04.430707+0000","last_clean":"2021-01-15T18:39:34.673184+0000","last_became_active":"2021-01-15T18:40:32.775060+0000","last_became_peered":"2021-01-15T18:40:32.775060+0000","last_unstale":"2021-01-15T18:46:04.430707+0000","last_undegraded":"2021-01-15T18:40:32.538928+0000","last_fullsized":"2021-01-15T18:40:32.515876+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"44'6527","reported_seq":"6793","reported_epoch":"44","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:45:55.058876+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:45:55.058876+0000","last_peered":"2021-01-15T18:45:55.058876+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:45:55.058876+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6527,"ondisk_log_size":6527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28795212,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9032","reported_epoch":"44","state":"active+clean","last_fresh":"2021-01-15T18:45:55.207885+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:45:55.207885+0000","last_peered":"2021-01-15T18:45:55.207885+0000","last_clean":"2021-01-15T18:45:55.207885+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:45:55.207885+0000","last_undegraded":"2021-01-15T18:45:55.207885+0000","last_fullsized":"2021-01-15T18:45:55.207885+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":446695724,"num_objects":54543,"num_object_clones":0,"num_object_copies":218172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100182,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11838,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893235200,"data_stored":893235200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100209,"ondisk_log_size":100209,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412224,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193791,"kb_avail":93959616,"statfs":{"total":96636764160,"available":96214646784,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198442036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":4.6059999999999999,"5min":3.6989999999999998,"15min":3.2389999999999999},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":22.928000000000001,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":4.4909999999999997,"5min":3.5550000000000002,"15min":3.496},"min":{"1min":0.52400000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":19.513000000000002,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.752},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.86199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:05 2021","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.2410000000000001,"15min":2.4100000000000001},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":13.433999999999999,"5min":15.164,"15min":15.164},"last":0.54400000000000004},{"interface":"front","average":{"1min":4.8730000000000002,"5min":4.1420000000000003,"15min":3.278},"min":{"1min":0.56899999999999995,"5min":0.503,"15min":0.503},"max":{"1min":17.222999999999999,"5min":18.300000000000001,"15min":18.300000000000001},"last":0.60299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412416,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959424,"statfs":{"total":96636764160,"available":96214450176,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.433},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.65300000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.58599999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416448,"kb_used_data":218424,"kb_used_omap":0,"kb_used_meta":198015,"kb_avail":93955392,"statfs":{"total":96636764160,"available":96210321408,"internally_reserved":0,"allocated":223666176,"data_stored":223432646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202767412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.435},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.57999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394348,"kb_used_data":218468,"kb_used_omap":0,"kb_used_meta":175871,"kb_avail":93977492,"statfs":{"total":96636764160,"available":96232951808,"internally_reserved":0,"allocated":223711232,"data_stored":223477702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180091956},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.85899999999999999}]}]}]}} 2021-01-15T18:46:10.179 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:10.601 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:46:10.600+0000 7f2f42559700 0 --2- 172.21.15.148:0/3364295251 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2f3c064a20 0x7f2f3c064e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-01-15T18:46:11.023 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:11.024 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9569/218172 objects degraded (4.386%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":421,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54543,"data_bytes":446695724,"bytes_used":1674686464,"bytes_avail":384872370176,"bytes_total":386547056640,"degraded_objects":9569,"degraded_total":218172,"degraded_ratio":0.043859890361732948,"recovering_objects_per_sec":80,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":642,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:11.042 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:11.043 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:12.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:12.401 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:12.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2021-01-15T18:46:10.555833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445491500,"num_objects":54396,"num_object_clones":0,"num_object_copies":217584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100329,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12425,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100356,"ondisk_log_size":100356,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1636352,"kb_used_data":873376,"kb_used_omap":2,"kb_used_meta":762877,"kb_avail":375851008,"statfs":{"total":386547056640,"available":384871432192,"internally_reserved":0,"allocated":894337024,"data_stored":893363756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":781186204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1204224,"num_objects":-147,"num_object_clones":0,"num_object_copies":-588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001451"},"pg_stats":[{"pgid":"1.f","version":"46'6129","reported_seq":"6393","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.913674+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:09.913674+0000","last_peered":"2021-01-15T18:46:09.913674+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:09.913674+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6129,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6172","reported_seq":"9102","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834787+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:09.834787+0000","last_peered":"2021-01-15T18:46:09.834787+0000","last_clean":"2021-01-15T18:46:09.834787+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:09.834787+0000","last_undegraded":"2021-01-15T18:46:09.834787+0000","last_fullsized":"2021-01-15T18:46:09.834787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26943654,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"44'6180","reported_seq":"7095","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:08.638610+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:08.638610+0000","last_peered":"2021-01-15T18:46:08.638610+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:08.638610+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27127915,"num_objects":3313,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6179,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2308}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6349","reported_seq":"6600","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.802881+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:09.802881+0000","last_peered":"2021-01-15T18:46:09.802881+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:09.802881+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6349,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205056,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6438","reported_seq":"7036","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.777724+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:09.777724+0000","last_peered":"2021-01-15T18:46:09.777724+0000","last_clean":"2021-01-15T18:46:09.777724+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:09.777724+0000","last_undegraded":"2021-01-15T18:46:09.777724+0000","last_fullsized":"2021-01-15T18:46:09.777724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6438,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27771046,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6140","reported_seq":"6357","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834878+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:09.834878+0000","last_peered":"2021-01-15T18:46:09.834878+0000","last_clean":"2021-01-15T18:46:09.834878+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:09.834878+0000","last_undegraded":"2021-01-15T18:46:09.834878+0000","last_fullsized":"2021-01-15T18:46:09.834878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6140,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738278,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6282","reported_seq":"6544","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.965086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:09.965086+0000","last_peered":"2021-01-15T18:46:09.965086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:09.965086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6282,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205222,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"44'6394","reported_seq":"6556","reported_epoch":"45","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:06.692262+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:06.692262+0000","last_peered":"2021-01-15T18:46:06.692262+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:06.692262+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6394,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28364883,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2050}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6326","reported_seq":"9267","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.793887+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:09.793887+0000","last_peered":"2021-01-15T18:46:09.793887+0000","last_clean":"2021-01-15T18:46:09.793887+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:09.793887+0000","last_undegraded":"2021-01-15T18:46:09.793887+0000","last_fullsized":"2021-01-15T18:46:09.793887+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6326,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27873363,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6335","reported_seq":"9282","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.854126+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:09.854126+0000","last_peered":"2021-01-15T18:46:09.854126+0000","last_clean":"2021-01-15T18:46:09.854126+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:09.854126+0000","last_undegraded":"2021-01-15T18:46:09.854126+0000","last_fullsized":"2021-01-15T18:46:09.854126+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6335,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28037369,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6332,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6176","reported_seq":"6422","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.620302+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:09.620302+0000","last_peered":"2021-01-15T18:46:09.620302+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:09.620302+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6176,"ondisk_log_size":6176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27672742,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6174,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6178","reported_seq":"9013","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.785101+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:09.785101+0000","last_peered":"2021-01-15T18:46:09.785101+0000","last_clean":"2021-01-15T18:46:09.785101+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:09.785101+0000","last_undegraded":"2021-01-15T18:46:09.785101+0000","last_fullsized":"2021-01-15T18:46:09.785101+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6178,"ondisk_log_size":6178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762854,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6160","reported_seq":"9061","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.267865+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:09.267865+0000","last_peered":"2021-01-15T18:46:09.267865+0000","last_clean":"2021-01-15T18:46:09.267865+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:09.267865+0000","last_undegraded":"2021-01-15T18:46:09.267865+0000","last_fullsized":"2021-01-15T18:46:09.267865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6160,"ondisk_log_size":6160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27156646,"num_objects":3316,"num_object_clones":0,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"6627","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2021-01-15T18:46:07.729433+0000","last_change":"2021-01-15T18:46:07.729433+0000","last_active":"2021-01-15T18:46:07.729433+0000","last_peered":"2021-01-15T18:46:07.729433+0000","last_clean":"2021-01-15T18:46:07.729433+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:07.729433+0000","last_undegraded":"2021-01-15T18:46:07.729433+0000","last_fullsized":"2021-01-15T18:46:07.729433+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6538","reported_seq":"6806","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.729006+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:09.729006+0000","last_peered":"2021-01-15T18:46:09.729006+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:09.729006+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6538,"ondisk_log_size":6538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28705100,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"44'6163","reported_seq":"9034","reported_epoch":"45","state":"active+clean","last_fresh":"2021-01-15T18:46:06.693335+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:06.693335+0000","last_peered":"2021-01-15T18:46:06.693335+0000","last_clean":"2021-01-15T18:46:06.693335+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:06.693335+0000","last_undegraded":"2021-01-15T18:46:06.693335+0000","last_fullsized":"2021-01-15T18:46:06.693335+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6163,"ondisk_log_size":6163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27656358,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":445491500,"num_objects":54396,"num_object_clones":0,"num_object_copies":217584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100329,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12425,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891445248,"data_stored":891445248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100356,"ondisk_log_size":100356,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412188,"kb_used_data":218160,"kb_used_omap":0,"kb_used_meta":193983,"kb_avail":93959652,"statfs":{"total":96636764160,"available":96214683648,"internally_reserved":0,"allocated":223395840,"data_stored":223152523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198638644},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412432,"kb_used_data":218440,"kb_used_omap":0,"kb_used_meta":193984,"kb_avail":93959408,"statfs":{"total":96636764160,"available":96214433792,"internally_reserved":0,"allocated":223682560,"data_stored":223439243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":198639616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.501}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.58899999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416412,"kb_used_data":218296,"kb_used_omap":0,"kb_used_meta":198079,"kb_avail":93955428,"statfs":{"total":96636764160,"available":96210358272,"internally_reserved":0,"allocated":223535104,"data_stored":223291787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202832948},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.66900000000000004},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.77200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.64000000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395320,"kb_used_data":218480,"kb_used_omap":0,"kb_used_meta":176831,"kb_avail":93976520,"statfs":{"total":96636764160,"available":96231956480,"internally_reserved":0,"allocated":223723520,"data_stored":223480203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181074996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.97899999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.093},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.2}]}]}]}} 2021-01-15T18:46:15.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:15.705 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:15.707 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:15.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2021-01-15T18:46:14.556673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":444959020,"num_objects":54331,"num_object_clones":0,"num_object_copies":217324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100394,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100421,"ondisk_log_size":100421,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1643500,"kb_used_data":871016,"kb_used_omap":2,"kb_used_meta":772413,"kb_avail":375843860,"statfs":{"total":386547056640,"available":384864112640,"internally_reserved":0,"allocated":891920384,"data_stored":890947116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":790951068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,19],"upper_bound":32768},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1736704,"num_objects":-212,"num_object_clones":0,"num_object_copies":-848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001584"},"pg_stats":[{"pgid":"1.f","version":"46'6129","reported_seq":"6393","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.913674+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:09.913674+0000","last_peered":"2021-01-15T18:46:09.913674+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:09.913674+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6129,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6172","reported_seq":"9102","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834787+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:09.834787+0000","last_peered":"2021-01-15T18:46:09.834787+0000","last_clean":"2021-01-15T18:46:09.834787+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:09.834787+0000","last_undegraded":"2021-01-15T18:46:09.834787+0000","last_fullsized":"2021-01-15T18:46:09.834787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26943654,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6196","reported_seq":"7915","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:13.637644+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:13.637644+0000","last_peered":"2021-01-15T18:46:13.637644+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:13.637644+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26996843,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6195,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2694}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6349","reported_seq":"6600","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.802881+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:09.802881+0000","last_peered":"2021-01-15T18:46:09.802881+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:09.802881+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6349,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205056,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6438","reported_seq":"7036","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.777724+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:09.777724+0000","last_peered":"2021-01-15T18:46:09.777724+0000","last_clean":"2021-01-15T18:46:09.777724+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:09.777724+0000","last_undegraded":"2021-01-15T18:46:09.777724+0000","last_fullsized":"2021-01-15T18:46:09.777724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6438,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27771046,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6140","reported_seq":"6357","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834878+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:09.834878+0000","last_peered":"2021-01-15T18:46:09.834878+0000","last_clean":"2021-01-15T18:46:09.834878+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:09.834878+0000","last_undegraded":"2021-01-15T18:46:09.834878+0000","last_fullsized":"2021-01-15T18:46:09.834878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6140,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738278,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6282","reported_seq":"6544","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.965086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:09.965086+0000","last_peered":"2021-01-15T18:46:09.965086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:09.965086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6282,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205222,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6408","reported_seq":"6570","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.526266+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:10.526266+0000","last_peered":"2021-01-15T18:46:10.526266+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:10.526266+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6408,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28250195,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6326","reported_seq":"9267","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.793887+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:09.793887+0000","last_peered":"2021-01-15T18:46:09.793887+0000","last_clean":"2021-01-15T18:46:09.793887+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:09.793887+0000","last_undegraded":"2021-01-15T18:46:09.793887+0000","last_fullsized":"2021-01-15T18:46:09.793887+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6326,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27873363,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6335","reported_seq":"9282","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.854126+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:09.854126+0000","last_peered":"2021-01-15T18:46:09.854126+0000","last_clean":"2021-01-15T18:46:09.854126+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:09.854126+0000","last_undegraded":"2021-01-15T18:46:09.854126+0000","last_fullsized":"2021-01-15T18:46:09.854126+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6335,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28037369,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6332,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6176","reported_seq":"6422","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.620302+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:09.620302+0000","last_peered":"2021-01-15T18:46:09.620302+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:09.620302+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6176,"ondisk_log_size":6176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27672742,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6174,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6178","reported_seq":"9013","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.785101+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:09.785101+0000","last_peered":"2021-01-15T18:46:09.785101+0000","last_clean":"2021-01-15T18:46:09.785101+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:09.785101+0000","last_undegraded":"2021-01-15T18:46:09.785101+0000","last_fullsized":"2021-01-15T18:46:09.785101+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6178,"ondisk_log_size":6178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762854,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6172","reported_seq":"9073","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:10.622938+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:10.622938+0000","last_peered":"2021-01-15T18:46:10.622938+0000","last_clean":"2021-01-15T18:46:10.622938+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:10.622938+0000","last_undegraded":"2021-01-15T18:46:10.622938+0000","last_fullsized":"2021-01-15T18:46:10.622938+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27058342,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"6627","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2021-01-15T18:46:07.729433+0000","last_change":"2021-01-15T18:46:07.729433+0000","last_active":"2021-01-15T18:46:07.729433+0000","last_peered":"2021-01-15T18:46:07.729433+0000","last_clean":"2021-01-15T18:46:07.729433+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:07.729433+0000","last_undegraded":"2021-01-15T18:46:07.729433+0000","last_fullsized":"2021-01-15T18:46:07.729433+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6538","reported_seq":"6806","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.729006+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:09.729006+0000","last_peered":"2021-01-15T18:46:09.729006+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:09.729006+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6538,"ondisk_log_size":6538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28705100,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6186","reported_seq":"9057","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.107709+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:11.107709+0000","last_peered":"2021-01-15T18:46:11.107709+0000","last_clean":"2021-01-15T18:46:11.107709+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:11.107709+0000","last_undegraded":"2021-01-15T18:46:11.107709+0000","last_fullsized":"2021-01-15T18:46:11.107709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6186,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27467942,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":444959020,"num_objects":54331,"num_object_clones":0,"num_object_copies":217324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100394,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":889229312,"data_stored":889229312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100421,"ondisk_log_size":100421,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412188,"kb_used_data":218160,"kb_used_omap":0,"kb_used_meta":193983,"kb_avail":93959652,"statfs":{"total":96636764160,"available":96214683648,"internally_reserved":0,"allocated":223395840,"data_stored":223152523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198638644},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415980,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":198784,"kb_avail":93955860,"statfs":{"total":96636764160,"available":96210800640,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":203554816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.62}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.64300000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420012,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":202815,"kb_avail":93951828,"statfs":{"total":96636764160,"available":96206671872,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":207682612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.434},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395320,"kb_used_data":218480,"kb_used_omap":0,"kb_used_meta":176831,"kb_avail":93976520,"statfs":{"total":96636764160,"available":96231956480,"internally_reserved":0,"allocated":223723520,"data_stored":223480203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181074996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.97899999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.093},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.2}]}]}]}} 2021-01-15T18:46:15.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:16.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:16.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8580/217324 objects degraded (3.948%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":426,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54331,"data_bytes":444959020,"bytes_used":1682944000,"bytes_avail":384864112640,"bytes_total":386547056640,"degraded_objects":8580,"degraded_total":217324,"degraded_ratio":0.039480223077064662,"recovering_objects_per_sec":123,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":26},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:16.091 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:16.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:16.382 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:16.384 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2021-01-15T18:46:14.556673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":444959020,"num_objects":54331,"num_object_clones":0,"num_object_copies":217324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100394,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100421,"ondisk_log_size":100421,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1643500,"kb_used_data":871016,"kb_used_omap":2,"kb_used_meta":772413,"kb_avail":375843860,"statfs":{"total":386547056640,"available":384864112640,"internally_reserved":0,"allocated":891920384,"data_stored":890947116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":790951068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,19],"upper_bound":32768},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1736704,"num_objects":-212,"num_object_clones":0,"num_object_copies":-848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001584"},"pg_stats":[{"pgid":"1.f","version":"46'6129","reported_seq":"6393","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.913674+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:09.913674+0000","last_peered":"2021-01-15T18:46:09.913674+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:09.913674+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6129,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27680768,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6172","reported_seq":"9102","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834787+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:09.834787+0000","last_peered":"2021-01-15T18:46:09.834787+0000","last_clean":"2021-01-15T18:46:09.834787+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:09.834787+0000","last_undegraded":"2021-01-15T18:46:09.834787+0000","last_fullsized":"2021-01-15T18:46:09.834787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26943654,"num_objects":3290,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6196","reported_seq":"7915","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:13.637644+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:13.637644+0000","last_peered":"2021-01-15T18:46:13.637644+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:13.637644+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26996843,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6195,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2694}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6349","reported_seq":"6600","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.802881+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:09.802881+0000","last_peered":"2021-01-15T18:46:09.802881+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:09.802881+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6349,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205056,"num_objects":3443,"num_object_clones":0,"num_object_copies":13772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6438","reported_seq":"7036","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.777724+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:09.777724+0000","last_peered":"2021-01-15T18:46:09.777724+0000","last_clean":"2021-01-15T18:46:09.777724+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:09.777724+0000","last_undegraded":"2021-01-15T18:46:09.777724+0000","last_fullsized":"2021-01-15T18:46:09.777724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6438,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27771046,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6140","reported_seq":"6357","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.834878+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:09.834878+0000","last_peered":"2021-01-15T18:46:09.834878+0000","last_clean":"2021-01-15T18:46:09.834878+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:09.834878+0000","last_undegraded":"2021-01-15T18:46:09.834878+0000","last_fullsized":"2021-01-15T18:46:09.834878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6140,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738278,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6282","reported_seq":"6544","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.965086+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:09.965086+0000","last_peered":"2021-01-15T18:46:09.965086+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:09.965086+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6282,"ondisk_log_size":6282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28205222,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6408","reported_seq":"6570","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.526266+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:10.526266+0000","last_peered":"2021-01-15T18:46:10.526266+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:10.526266+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6408,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28250195,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6326","reported_seq":"9267","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.793887+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:09.793887+0000","last_peered":"2021-01-15T18:46:09.793887+0000","last_clean":"2021-01-15T18:46:09.793887+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:09.793887+0000","last_undegraded":"2021-01-15T18:46:09.793887+0000","last_fullsized":"2021-01-15T18:46:09.793887+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6326,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27873363,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6335","reported_seq":"9282","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.854126+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:09.854126+0000","last_peered":"2021-01-15T18:46:09.854126+0000","last_clean":"2021-01-15T18:46:09.854126+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:09.854126+0000","last_undegraded":"2021-01-15T18:46:09.854126+0000","last_fullsized":"2021-01-15T18:46:09.854126+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6335,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28037369,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6332,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6176","reported_seq":"6422","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.620302+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:09.620302+0000","last_peered":"2021-01-15T18:46:09.620302+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:09.620302+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6176,"ondisk_log_size":6176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27672742,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6174,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6178","reported_seq":"9013","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:09.785101+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:09.785101+0000","last_peered":"2021-01-15T18:46:09.785101+0000","last_clean":"2021-01-15T18:46:09.785101+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:09.785101+0000","last_undegraded":"2021-01-15T18:46:09.785101+0000","last_fullsized":"2021-01-15T18:46:09.785101+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6178,"ondisk_log_size":6178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27762854,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6172","reported_seq":"9073","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:10.622938+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:10.622938+0000","last_peered":"2021-01-15T18:46:10.622938+0000","last_clean":"2021-01-15T18:46:10.622938+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:10.622938+0000","last_undegraded":"2021-01-15T18:46:10.622938+0000","last_fullsized":"2021-01-15T18:46:10.622938+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27058342,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"6627","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2021-01-15T18:46:07.729433+0000","last_change":"2021-01-15T18:46:07.729433+0000","last_active":"2021-01-15T18:46:07.729433+0000","last_peered":"2021-01-15T18:46:07.729433+0000","last_clean":"2021-01-15T18:46:07.729433+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:07.729433+0000","last_undegraded":"2021-01-15T18:46:07.729433+0000","last_fullsized":"2021-01-15T18:46:07.729433+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6538","reported_seq":"6806","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:09.729006+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:09.729006+0000","last_peered":"2021-01-15T18:46:09.729006+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:09.729006+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6538,"ondisk_log_size":6538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28705100,"num_objects":3506,"num_object_clones":0,"num_object_copies":14024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6186","reported_seq":"9057","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.107709+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:11.107709+0000","last_peered":"2021-01-15T18:46:11.107709+0000","last_clean":"2021-01-15T18:46:11.107709+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:11.107709+0000","last_undegraded":"2021-01-15T18:46:11.107709+0000","last_fullsized":"2021-01-15T18:46:11.107709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6186,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27467942,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":444959020,"num_objects":54331,"num_object_clones":0,"num_object_copies":217324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100394,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":889229312,"data_stored":889229312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100421,"ondisk_log_size":100421,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215184,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412188,"kb_used_data":218160,"kb_used_omap":0,"kb_used_meta":193983,"kb_avail":93959652,"statfs":{"total":96636764160,"available":96214683648,"internally_reserved":0,"allocated":223395840,"data_stored":223152523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":198638644},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415980,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":198784,"kb_avail":93955860,"statfs":{"total":96636764160,"available":96210800640,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":203554816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.62}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.64300000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420012,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":202815,"kb_avail":93951828,"statfs":{"total":96636764160,"available":96206671872,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":207682612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.434},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395320,"kb_used_data":218480,"kb_used_omap":0,"kb_used_meta":176831,"kb_avail":93976520,"statfs":{"total":96636764160,"available":96231956480,"internally_reserved":0,"allocated":223723520,"data_stored":223480203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":181074996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.97899999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.093},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.2}]}]}]}} 2021-01-15T18:46:19.397 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:20.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:20.347 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:20.362 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2021-01-15T18:46:18.557423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":444131628,"num_objects":54230,"num_object_clones":0,"num_object_copies":216920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100495,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100522,"ondisk_log_size":100522,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1655320,"kb_used_data":868792,"kb_used_omap":2,"kb_used_meta":786493,"kb_avail":375832040,"statfs":{"total":386547056640,"available":384852008960,"internally_reserved":0,"allocated":889643008,"data_stored":888669740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":805368988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1,3,36,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2564096,"num_objects":-313,"num_object_clones":0,"num_object_copies":-1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.869416"},"pg_stats":[{"pgid":"1.f","version":"46'6134","reported_seq":"6398","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.833001+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:10.833001+0000","last_peered":"2021-01-15T18:46:10.833001+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:10.833001+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6134,"ondisk_log_size":6134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27639808,"num_objects":3374,"num_object_clones":0,"num_object_copies":13496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6184","reported_seq":"9114","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.029889+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:11.029889+0000","last_peered":"2021-01-15T18:46:11.029889+0000","last_clean":"2021-01-15T18:46:11.029889+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:11.029889+0000","last_undegraded":"2021-01-15T18:46:11.029889+0000","last_fullsized":"2021-01-15T18:46:11.029889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6184,"ondisk_log_size":6184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26845350,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6196","reported_seq":"7915","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:13.637644+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:13.637644+0000","last_peered":"2021-01-15T18:46:13.637644+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:13.637644+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6196,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26996843,"num_objects":3297,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6195,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2694}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6360","reported_seq":"6612","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.956256+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:10.956256+0000","last_peered":"2021-01-15T18:46:10.956256+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:10.956256+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6360,"ondisk_log_size":6360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28114944,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6360,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6449","reported_seq":"7047","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.062498+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:11.062498+0000","last_peered":"2021-01-15T18:46:11.062498+0000","last_clean":"2021-01-15T18:46:11.062498+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:11.062498+0000","last_undegraded":"2021-01-15T18:46:11.062498+0000","last_fullsized":"2021-01-15T18:46:11.062498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6449,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27680934,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6152","reported_seq":"6369","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:10.974390+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:10.974390+0000","last_peered":"2021-01-15T18:46:10.974390+0000","last_clean":"2021-01-15T18:46:10.974390+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:10.974390+0000","last_undegraded":"2021-01-15T18:46:10.974390+0000","last_fullsized":"2021-01-15T18:46:10.974390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6152,"ondisk_log_size":6152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27639974,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6286","reported_seq":"6548","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.299756+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:10.299756+0000","last_peered":"2021-01-15T18:46:10.299756+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:10.299756+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6286,"ondisk_log_size":6286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28172454,"num_objects":3440,"num_object_clones":0,"num_object_copies":13760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6408","reported_seq":"6570","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.526266+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:10.526266+0000","last_peered":"2021-01-15T18:46:10.526266+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:10.526266+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6408,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28250195,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6335","reported_seq":"9276","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.085500+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:11.085500+0000","last_peered":"2021-01-15T18:46:11.085500+0000","last_clean":"2021-01-15T18:46:11.085500+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:11.085500+0000","last_undegraded":"2021-01-15T18:46:11.085500+0000","last_fullsized":"2021-01-15T18:46:11.085500+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6335,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27799635,"num_objects":3394,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6340","reported_seq":"9287","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:10.818102+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:10.818102+0000","last_peered":"2021-01-15T18:46:10.818102+0000","last_clean":"2021-01-15T18:46:10.818102+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:10.818102+0000","last_undegraded":"2021-01-15T18:46:10.818102+0000","last_fullsized":"2021-01-15T18:46:10.818102+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6340,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27996409,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6337,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6185","reported_seq":"6431","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.930406+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:10.930406+0000","last_peered":"2021-01-15T18:46:10.930406+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:10.930406+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6185,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27599014,"num_objects":3370,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2078}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6193","reported_seq":"9028","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.123179+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:11.123179+0000","last_peered":"2021-01-15T18:46:11.123179+0000","last_clean":"2021-01-15T18:46:11.123179+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:11.123179+0000","last_undegraded":"2021-01-15T18:46:11.123179+0000","last_fullsized":"2021-01-15T18:46:11.123179+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6193,"ondisk_log_size":6193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27639974,"num_objects":3375,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6172","reported_seq":"9073","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:10.622938+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:10.622938+0000","last_peered":"2021-01-15T18:46:10.622938+0000","last_clean":"2021-01-15T18:46:10.622938+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:10.622938+0000","last_undegraded":"2021-01-15T18:46:10.622938+0000","last_fullsized":"2021-01-15T18:46:10.622938+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6172,"ondisk_log_size":6172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27058342,"num_objects":3304,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"43'6396","reported_seq":"6627","reported_epoch":"46","state":"active+clean+wait","last_fresh":"2021-01-15T18:46:07.729433+0000","last_change":"2021-01-15T18:46:07.729433+0000","last_active":"2021-01-15T18:46:07.729433+0000","last_peered":"2021-01-15T18:46:07.729433+0000","last_clean":"2021-01-15T18:46:07.729433+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:07.729433+0000","last_undegraded":"2021-01-15T18:46:07.729433+0000","last_fullsized":"2021-01-15T18:46:07.729433+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28590246,"num_objects":3491,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6394,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6546","reported_seq":"6814","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:10.699166+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:10.699166+0000","last_peered":"2021-01-15T18:46:10.699166+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:10.699166+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6546,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28639564,"num_objects":3498,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6543,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6186","reported_seq":"9057","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:11.107709+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:11.107709+0000","last_peered":"2021-01-15T18:46:11.107709+0000","last_clean":"2021-01-15T18:46:11.107709+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:11.107709+0000","last_undegraded":"2021-01-15T18:46:11.107709+0000","last_fullsized":"2021-01-15T18:46:11.107709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6186,"ondisk_log_size":6186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27467942,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":444131628,"num_objects":54230,"num_object_clones":0,"num_object_copies":216920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100495,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":888102912,"data_stored":888102912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100522,"ondisk_log_size":100522,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415852,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":198655,"kb_avail":93955988,"statfs":{"total":96636764160,"available":96210931712,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":203422772},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.627},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":0.76700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415980,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":198784,"kb_avail":93955860,"statfs":{"total":96636764160,"available":96210800640,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":203554816},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.62}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.64300000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420012,"kb_used_data":217188,"kb_used_omap":0,"kb_used_meta":202815,"kb_avail":93951828,"statfs":{"total":96636764160,"available":96206671872,"internally_reserved":0,"allocated":222400512,"data_stored":222157195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":207682612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.434},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403476,"kb_used_data":217228,"kb_used_omap":0,"kb_used_meta":186239,"kb_avail":93968364,"statfs":{"total":96636764160,"available":96223604736,"internally_reserved":0,"allocated":222441472,"data_stored":222198155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":190708788},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1,3,36],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.68200000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.0469999999999999}]}]}]}} 2021-01-15T18:46:20.363 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:21.785 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:21.786 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8104/214628 objects degraded (3.776%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":432,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53657,"data_bytes":439437612,"bytes_used":1688186880,"bytes_avail":384858869760,"bytes_total":386547056640,"degraded_objects":8104,"degraded_total":214628,"degraded_ratio":0.037758353989227875,"recovering_objects_per_sec":122,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1465,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":73},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:21.849 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:21.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:22.702 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:22.766 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2021-01-15T18:46:22.558309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439437612,"num_objects":53657,"num_object_clones":0,"num_object_copies":214628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101068,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13303,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101095,"ondisk_log_size":101095,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1648620,"kb_used_data":862772,"kb_used_omap":2,"kb_used_meta":785725,"kb_avail":375838740,"statfs":{"total":386547056640,"available":384858869760,"internally_reserved":0,"allocated":883478528,"data_stored":882505260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":804582556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,40,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6053888,"num_objects":-739,"num_object_clones":0,"num_object_copies":-2956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":878,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002475"},"pg_stats":[{"pgid":"1.f","version":"46'6180","reported_seq":"6444","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:19.959606+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:19.959606+0000","last_peered":"2021-01-15T18:46:19.959606+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:19.959606+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6180,"ondisk_log_size":6180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27262976,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6180,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2078}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6219","reported_seq":"9148","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.830459+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:19.830459+0000","last_peered":"2021-01-15T18:46:19.830459+0000","last_clean":"2021-01-15T18:46:19.830459+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:19.830459+0000","last_undegraded":"2021-01-15T18:46:19.830459+0000","last_fullsized":"2021-01-15T18:46:19.830459+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6219,"ondisk_log_size":6219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26558630,"num_objects":3243,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6217,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6212","reported_seq":"8879","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:18.549043+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:18.549043+0000","last_peered":"2021-01-15T18:46:18.549043+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:18.549043+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6212,"ondisk_log_size":6212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26865771,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6211,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1289,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6396","reported_seq":"6648","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:19.904684+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:19.904684+0000","last_peered":"2021-01-15T18:46:19.904684+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:19.904684+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6396,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27820032,"num_objects":3396,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6489","reported_seq":"7087","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.994250+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:19.994250+0000","last_peered":"2021-01-15T18:46:19.994250+0000","last_clean":"2021-01-15T18:46:19.994250+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:19.994250+0000","last_undegraded":"2021-01-15T18:46:19.994250+0000","last_fullsized":"2021-01-15T18:46:19.994250+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6489,"ondisk_log_size":6489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27353254,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6187","reported_seq":"6404","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.859659+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:19.859659+0000","last_peered":"2021-01-15T18:46:19.859659+0000","last_clean":"2021-01-15T18:46:19.859659+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:19.859659+0000","last_undegraded":"2021-01-15T18:46:19.859659+0000","last_fullsized":"2021-01-15T18:46:19.859659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6187,"ondisk_log_size":6187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27353254,"num_objects":3340,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6323","reported_seq":"6585","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:19.960150+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:19.960150+0000","last_peered":"2021-01-15T18:46:19.960150+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:19.960150+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6323,"ondisk_log_size":6323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27869350,"num_objects":3403,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2110}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6421","reported_seq":"6583","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:18.486028+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:18.486028+0000","last_peered":"2021-01-15T18:46:18.486028+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:18.486028+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6421,"ondisk_log_size":6421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28143699,"num_objects":3436,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2023}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6379","reported_seq":"9320","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.787783+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:19.787783+0000","last_peered":"2021-01-15T18:46:19.787783+0000","last_clean":"2021-01-15T18:46:19.787783+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:19.787783+0000","last_undegraded":"2021-01-15T18:46:19.787783+0000","last_fullsized":"2021-01-15T18:46:19.787783+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6379,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27439187,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6378,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6382","reported_seq":"9329","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.948337+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:19.948337+0000","last_peered":"2021-01-15T18:46:19.948337+0000","last_clean":"2021-01-15T18:46:19.948337+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:19.948337+0000","last_undegraded":"2021-01-15T18:46:19.948337+0000","last_fullsized":"2021-01-15T18:46:19.948337+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6382,"ondisk_log_size":6382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27652345,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6379,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6220","reported_seq":"6466","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:19.908470+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:19.908470+0000","last_peered":"2021-01-15T18:46:19.908470+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:19.908470+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6220,"ondisk_log_size":6220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27312294,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6224","reported_seq":"9059","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.999993+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:19.999993+0000","last_peered":"2021-01-15T18:46:19.999993+0000","last_clean":"2021-01-15T18:46:19.999993+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:19.999993+0000","last_undegraded":"2021-01-15T18:46:19.999993+0000","last_fullsized":"2021-01-15T18:46:19.999993+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6224,"ondisk_log_size":6224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27386022,"num_objects":3344,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6222,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6201","reported_seq":"9102","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.372926+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:19.372926+0000","last_peered":"2021-01-15T18:46:19.372926+0000","last_clean":"2021-01-15T18:46:19.372926+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:19.372926+0000","last_undegraded":"2021-01-15T18:46:19.372926+0000","last_fullsized":"2021-01-15T18:46:19.372926+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6201,"ondisk_log_size":6201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26820774,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"46'6452","reported_seq":"6684","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:19.982108+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:19.982108+0000","last_peered":"2021-01-15T18:46:19.982108+0000","last_clean":"2021-01-15T18:46:19.982108+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:19.982108+0000","last_undegraded":"2021-01-15T18:46:19.982108+0000","last_fullsized":"2021-01-15T18:46:19.982108+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6452,"ondisk_log_size":6452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28131494,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6595","reported_seq":"6867","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:19.724810+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:19.724810+0000","last_peered":"2021-01-15T18:46:19.724810+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:19.724810+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6595,"ondisk_log_size":6595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28238156,"num_objects":3449,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6215","reported_seq":"9086","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:18.625483+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:18.625483+0000","last_peered":"2021-01-15T18:46:18.625483+0000","last_clean":"2021-01-15T18:46:18.625483+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:18.625483+0000","last_undegraded":"2021-01-15T18:46:18.625483+0000","last_fullsized":"2021-01-15T18:46:18.625483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6215,"ondisk_log_size":6215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27230374,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":439437612,"num_objects":53657,"num_object_clones":0,"num_object_copies":214628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101068,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13303,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":879583232,"data_stored":879583232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101095,"ondisk_log_size":101095,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215186,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418912,"kb_used_data":215216,"kb_used_omap":0,"kb_used_meta":203647,"kb_avail":93952928,"statfs":{"total":96636764160,"available":96207798272,"internally_reserved":0,"allocated":220381184,"data_stored":220137867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208534580},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":65536},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":1},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.1220000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.4850000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.2509999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415760,"kb_used_data":216712,"kb_used_omap":0,"kb_used_meta":199040,"kb_avail":93956080,"statfs":{"total":96636764160,"available":96211025920,"internally_reserved":0,"allocated":221913088,"data_stored":221669771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":203816960},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.79000000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.59999999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.65300000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406764,"kb_used_data":215368,"kb_used_omap":0,"kb_used_meta":191359,"kb_avail":93965076,"statfs":{"total":96636764160,"available":96220237824,"internally_reserved":0,"allocated":220536832,"data_stored":220293515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195951668},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.434},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.70999999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407184,"kb_used_data":215476,"kb_used_omap":0,"kb_used_meta":191679,"kb_avail":93964656,"statfs":{"total":96636764160,"available":96219807744,"internally_reserved":0,"allocated":220647424,"data_stored":220404107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":196279348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,40],"upper_bound":16384},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.86699999999999999}]}]}]}} 2021-01-15T18:46:25.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:27.071 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:27.158 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2021-01-15T18:46:26.559123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429599020,"num_objects":52456,"num_object_clones":0,"num_object_copies":209824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102269,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13365,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102296,"ondisk_log_size":102296,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1622316,"kb_used_data":843944,"kb_used_omap":2,"kb_used_meta":778237,"kb_avail":375865044,"statfs":{"total":386547056640,"available":384885805056,"internally_reserved":0,"allocated":864198656,"data_stored":863225388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":796914844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,4,13,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15360000,"num_objects":-1875,"num_object_clones":0,"num_object_copies":-7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002449"},"pg_stats":[{"pgid":"1.f","version":"46'6240","reported_seq":"6504","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.734507+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:24.734507+0000","last_peered":"2021-01-15T18:46:24.734507+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:24.734507+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6240,"ondisk_log_size":6240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26771456,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6289","reported_seq":"9219","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.992672+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:24.992672+0000","last_peered":"2021-01-15T18:46:24.992672+0000","last_clean":"2021-01-15T18:46:24.992672+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:24.992672+0000","last_undegraded":"2021-01-15T18:46:24.992672+0000","last_fullsized":"2021-01-15T18:46:24.992672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6289,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25985190,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6287","reported_seq":"9077","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:23.617092+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:23.617092+0000","last_peered":"2021-01-15T18:46:23.617092+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:23.617092+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6287,"ondisk_log_size":6287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26251371,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6286,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6471","reported_seq":"6723","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.838320+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:24.838320+0000","last_peered":"2021-01-15T18:46:24.838320+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:24.838320+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6471,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27205632,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6555","reported_seq":"7153","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.821110+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:24.821110+0000","last_peered":"2021-01-15T18:46:24.821110+0000","last_clean":"2021-01-15T18:46:24.821110+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:24.821110+0000","last_undegraded":"2021-01-15T18:46:24.821110+0000","last_fullsized":"2021-01-15T18:46:24.821110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6555,"ondisk_log_size":6555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26812582,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6251","reported_seq":"6468","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.992704+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:24.992704+0000","last_peered":"2021-01-15T18:46:24.992704+0000","last_clean":"2021-01-15T18:46:24.992704+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:24.992704+0000","last_undegraded":"2021-01-15T18:46:24.992704+0000","last_fullsized":"2021-01-15T18:46:24.992704+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6251,"ondisk_log_size":6251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26828966,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6387","reported_seq":"6649","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.650023+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:24.650023+0000","last_peered":"2021-01-15T18:46:24.650023+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:24.650023+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6387,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27345062,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6385,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2046}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6501","reported_seq":"6663","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:23.490110+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:23.490110+0000","last_peered":"2021-01-15T18:46:23.490110+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:23.490110+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6501,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27488339,"num_objects":3356,"num_object_clones":0,"num_object_copies":13424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6500,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6461","reported_seq":"9402","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.902168+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:24.902168+0000","last_peered":"2021-01-15T18:46:24.902168+0000","last_clean":"2021-01-15T18:46:24.902168+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:24.902168+0000","last_undegraded":"2021-01-15T18:46:24.902168+0000","last_fullsized":"2021-01-15T18:46:24.902168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6461,"ondisk_log_size":6461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26767443,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6466","reported_seq":"9413","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.926037+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:24.926037+0000","last_peered":"2021-01-15T18:46:24.926037+0000","last_clean":"2021-01-15T18:46:24.926037+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:24.926037+0000","last_undegraded":"2021-01-15T18:46:24.926037+0000","last_fullsized":"2021-01-15T18:46:24.926037+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6466,"ondisk_log_size":6466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26964217,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6300","reported_seq":"6545","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.901880+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:24.901880+0000","last_peered":"2021-01-15T18:46:24.901880+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:24.901880+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6300,"ondisk_log_size":6300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26656934,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6298","reported_seq":"9133","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.901829+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:24.901829+0000","last_peered":"2021-01-15T18:46:24.901829+0000","last_clean":"2021-01-15T18:46:24.901829+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:24.901829+0000","last_undegraded":"2021-01-15T18:46:24.901829+0000","last_fullsized":"2021-01-15T18:46:24.901829+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6298,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26779814,"num_objects":3270,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6266","reported_seq":"9167","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.338979+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:24.338979+0000","last_peered":"2021-01-15T18:46:24.338979+0000","last_clean":"2021-01-15T18:46:24.338979+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:24.338979+0000","last_undegraded":"2021-01-15T18:46:24.338979+0000","last_fullsized":"2021-01-15T18:46:24.338979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6266,"ondisk_log_size":6266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26288294,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6264,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"46'6544","reported_seq":"6776","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.994723+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:24.994723+0000","last_peered":"2021-01-15T18:46:24.994723+0000","last_clean":"2021-01-15T18:46:24.994723+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:24.994723+0000","last_undegraded":"2021-01-15T18:46:24.994723+0000","last_fullsized":"2021-01-15T18:46:24.994723+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6544,"ondisk_log_size":6544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27377830,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6693","reported_seq":"6965","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.977488+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:24.977488+0000","last_peered":"2021-01-15T18:46:24.977488+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:24.977488+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6693,"ondisk_log_size":6693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27435340,"num_objects":3351,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6690,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6287","reported_seq":"9158","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:23.557403+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:23.557403+0000","last_peered":"2021-01-15T18:46:23.557403+0000","last_clean":"2021-01-15T18:46:23.557403+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:23.557403+0000","last_undegraded":"2021-01-15T18:46:23.557403+0000","last_fullsized":"2021-01-15T18:46:23.557403+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6287,"ondisk_log_size":6287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26640550,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6285,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":429599020,"num_objects":52456,"num_object_clones":0,"num_object_copies":209824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102269,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13365,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":859914240,"data_stored":859914240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102296,"ondisk_log_size":102296,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408188,"kb_used_data":210520,"kb_used_omap":0,"kb_used_meta":197631,"kb_avail":93963652,"statfs":{"total":96636764160,"available":96218779648,"internally_reserved":0,"allocated":215572480,"data_stored":215329163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202374196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":1.8069999999999999},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.9159999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.0350000000000001},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.137}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":18.959},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":14.429}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405112,"kb_used_data":212036,"kb_used_omap":0,"kb_used_meta":193024,"kb_avail":93966728,"statfs":{"total":96636764160,"available":96221929472,"internally_reserved":0,"allocated":217124864,"data_stored":216881547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":197656576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.681},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.84299999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":13.586},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.65100000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":1.798},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.94199999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412436,"kb_used_data":210632,"kb_used_omap":0,"kb_used_meta":201791,"kb_avail":93959404,"statfs":{"total":96636764160,"available":96214429696,"internally_reserved":0,"allocated":215687168,"data_stored":215443851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":206634036},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":5.4029999999999996},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":4.3029999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.56999999999999995},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.4509999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":7.5119999999999996},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":12.944000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396580,"kb_used_data":210756,"kb_used_omap":0,"kb_used_meta":185791,"kb_avail":93975260,"statfs":{"total":96636764160,"available":96230666240,"internally_reserved":0,"allocated":215814144,"data_stored":215570827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":190250036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":10.673999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":1.9259999999999999},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.64000000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":11.231999999999999},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":11.332000000000001}]}]}]}} 2021-01-15T18:46:27.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:28.610 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:28.611 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8042/209824 objects degraded (3.833%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":439,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52456,"data_bytes":429599020,"bytes_used":1661251584,"bytes_avail":384885805056,"bytes_total":386547056640,"degraded_objects":8042,"degraded_total":209824,"degraded_ratio":0.038327360073204211,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":156},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:28.659 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:28.660 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:29.723 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2021-01-15T18:46:29.611983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":427116844,"num_objects":52153,"num_object_clones":0,"num_object_copies":208612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102572,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13430,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102599,"ondisk_log_size":102599,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1617044,"kb_used_data":834292,"kb_used_omap":2,"kb_used_meta":782589,"kb_avail":375870316,"statfs":{"total":386547056640,"available":384891203584,"internally_reserved":0,"allocated":854315008,"data_stored":853341740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":801371292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,2,13,3,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":120,"apply_latency_ms":120,"commit_latency_ns":120000000,"apply_latency_ns":120000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17014784,"num_objects":-2077,"num_object_clones":0,"num_object_copies":-8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2077,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.054560"},"pg_stats":[{"pgid":"1.f","version":"46'6240","reported_seq":"6504","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.734507+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:24.734507+0000","last_peered":"2021-01-15T18:46:24.734507+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:24.734507+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6240,"ondisk_log_size":6240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26771456,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"46'6289","reported_seq":"9219","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.992672+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:24.992672+0000","last_peered":"2021-01-15T18:46:24.992672+0000","last_clean":"2021-01-15T18:46:24.992672+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:24.992672+0000","last_undegraded":"2021-01-15T18:46:24.992672+0000","last_fullsized":"2021-01-15T18:46:24.992672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6289,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25985190,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6352","reported_seq":"9273","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:28.604801+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:28.604801+0000","last_peered":"2021-01-15T18:46:28.604801+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:28.604801+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6352,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25718891,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6351,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"46'6471","reported_seq":"6723","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.838320+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:24.838320+0000","last_peered":"2021-01-15T18:46:24.838320+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:24.838320+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6471,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27205632,"num_objects":3321,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"46'6555","reported_seq":"7153","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.821110+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:24.821110+0000","last_peered":"2021-01-15T18:46:24.821110+0000","last_clean":"2021-01-15T18:46:24.821110+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:24.821110+0000","last_undegraded":"2021-01-15T18:46:24.821110+0000","last_fullsized":"2021-01-15T18:46:24.821110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6555,"ondisk_log_size":6555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26812582,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"46'6251","reported_seq":"6468","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.992704+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:24.992704+0000","last_peered":"2021-01-15T18:46:24.992704+0000","last_clean":"2021-01-15T18:46:24.992704+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:24.992704+0000","last_undegraded":"2021-01-15T18:46:24.992704+0000","last_fullsized":"2021-01-15T18:46:24.992704+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6251,"ondisk_log_size":6251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26828966,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"46'6387","reported_seq":"6649","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.650023+0000","last_change":"2021-01-15T18:40:32.755528+0000","last_active":"2021-01-15T18:46:24.650023+0000","last_peered":"2021-01-15T18:46:24.650023+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:24.650023+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6387,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27345062,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6385,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2046}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6596","reported_seq":"6758","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:28.521195+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:28.521195+0000","last_peered":"2021-01-15T18:46:28.521195+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:28.521195+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6596,"ondisk_log_size":6596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26710099,"num_objects":3261,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6595,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"46'6461","reported_seq":"9402","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.902168+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:24.902168+0000","last_peered":"2021-01-15T18:46:24.902168+0000","last_clean":"2021-01-15T18:46:24.902168+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:24.902168+0000","last_undegraded":"2021-01-15T18:46:24.902168+0000","last_fullsized":"2021-01-15T18:46:24.902168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6461,"ondisk_log_size":6461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26767443,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"46'6466","reported_seq":"9413","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.926037+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:24.926037+0000","last_peered":"2021-01-15T18:46:24.926037+0000","last_clean":"2021-01-15T18:46:24.926037+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:24.926037+0000","last_undegraded":"2021-01-15T18:46:24.926037+0000","last_fullsized":"2021-01-15T18:46:24.926037+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6466,"ondisk_log_size":6466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26964217,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"46'6300","reported_seq":"6545","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.901880+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:24.901880+0000","last_peered":"2021-01-15T18:46:24.901880+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:24.901880+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6300,"ondisk_log_size":6300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26656934,"num_objects":3255,"num_object_clones":0,"num_object_copies":13020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6298,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"46'6298","reported_seq":"9133","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.901829+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:24.901829+0000","last_peered":"2021-01-15T18:46:24.901829+0000","last_clean":"2021-01-15T18:46:24.901829+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:24.901829+0000","last_undegraded":"2021-01-15T18:46:24.901829+0000","last_fullsized":"2021-01-15T18:46:24.901829+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6298,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26779814,"num_objects":3270,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6349","reported_seq":"9250","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:29.358982+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:29.358982+0000","last_peered":"2021-01-15T18:46:29.358982+0000","last_clean":"2021-01-15T18:46:29.358982+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:29.358982+0000","last_undegraded":"2021-01-15T18:46:29.358982+0000","last_fullsized":"2021-01-15T18:46:29.358982+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6349,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25608358,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"46'6544","reported_seq":"6776","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:24.994723+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:24.994723+0000","last_peered":"2021-01-15T18:46:24.994723+0000","last_clean":"2021-01-15T18:46:24.994723+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:24.994723+0000","last_undegraded":"2021-01-15T18:46:24.994723+0000","last_fullsized":"2021-01-15T18:46:24.994723+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6544,"ondisk_log_size":6544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27377830,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"46'6693","reported_seq":"6965","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:24.977488+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:24.977488+0000","last_peered":"2021-01-15T18:46:24.977488+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:24.977488+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6693,"ondisk_log_size":6693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27435340,"num_objects":3351,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6690,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6347","reported_seq":"9218","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:28.384230+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:28.384230+0000","last_peered":"2021-01-15T18:46:28.384230+0000","last_clean":"2021-01-15T18:46:28.384230+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:28.384230+0000","last_undegraded":"2021-01-15T18:46:28.384230+0000","last_fullsized":"2021-01-15T18:46:28.384230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6347,"ondisk_log_size":6347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26149030,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":427116844,"num_objects":52153,"num_object_clones":0,"num_object_copies":208612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102572,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13430,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":850108416,"data_stored":850108416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102599,"ondisk_log_size":102599,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215187,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408188,"kb_used_data":210520,"kb_used_omap":0,"kb_used_meta":197631,"kb_avail":93963652,"statfs":{"total":96636764160,"available":96218779648,"internally_reserved":0,"allocated":215572480,"data_stored":215329163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202374196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":1.8069999999999999},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.9159999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.0350000000000001},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.137}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":18.959},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":14.429}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410628,"kb_used_data":207192,"kb_used_omap":0,"kb_used_meta":203392,"kb_avail":93961212,"statfs":{"total":96636764160,"available":96216281088,"internally_reserved":0,"allocated":212164608,"data_stored":211921291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":208273408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.873}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.1839999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401648,"kb_used_data":205824,"kb_used_omap":0,"kb_used_meta":195775,"kb_avail":93970192,"statfs":{"total":96636764160,"available":96225476608,"internally_reserved":0,"allocated":210763776,"data_stored":210520459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":200473652},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":2.5379999999999998},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":6.1959999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":1.0820000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.131}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":1.7070000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396580,"kb_used_data":210756,"kb_used_omap":0,"kb_used_meta":185791,"kb_avail":93975260,"statfs":{"total":96636764160,"available":96230666240,"internally_reserved":0,"allocated":215814144,"data_stored":215570827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":190250036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":10.673999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":1.9259999999999999},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":0.64000000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":11.231999999999999},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":11.332000000000001}]}]}]}} 2021-01-15T18:46:32.794 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:34.009 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:34.059 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2021-01-15T18:46:32.561155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":419940652,"num_objects":51277,"num_object_clones":0,"num_object_copies":205108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103448,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13450,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103475,"ondisk_log_size":103475,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1620048,"kb_used_data":824640,"kb_used_omap":2,"kb_used_meta":795261,"kb_avail":375867312,"statfs":{"total":386547056640,"available":384888127488,"internally_reserved":0,"allocated":844431360,"data_stored":843458092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":814347420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,21,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":116,"apply_latency_ms":116,"commit_latency_ns":116000000,"apply_latency_ns":116000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17104896,"num_objects":-2088,"num_object_clones":0,"num_object_copies":-8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002488"},"pg_stats":[{"pgid":"1.f","version":"47'6311","reported_seq":"6576","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:29.865872+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:29.865872+0000","last_peered":"2021-01-15T18:46:29.865872+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:29.865872+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6311,"ondisk_log_size":6311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"47'6360","reported_seq":"9290","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:30.035697+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:30.035697+0000","last_peered":"2021-01-15T18:46:30.035697+0000","last_clean":"2021-01-15T18:46:30.035697+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:30.035697+0000","last_undegraded":"2021-01-15T18:46:30.035697+0000","last_fullsized":"2021-01-15T18:46:30.035697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6360,"ondisk_log_size":6360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25403558,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"46'6352","reported_seq":"9273","reported_epoch":"46","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:28.604801+0000","last_change":"2021-01-15T18:46:05.981289+0000","last_active":"2021-01-15T18:46:28.604801+0000","last_peered":"2021-01-15T18:46:28.604801+0000","last_clean":"2021-01-15T18:39:34.588836+0000","last_became_active":"2021-01-15T18:40:32.762656+0000","last_became_peered":"2021-01-15T18:40:32.762656+0000","last_unstale":"2021-01-15T18:46:28.604801+0000","last_undegraded":"2021-01-15T18:40:32.538714+0000","last_fullsized":"2021-01-15T18:40:32.527046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6352,"ondisk_log_size":6352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25718891,"num_objects":3141,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6351,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"47'6540","reported_seq":"6793","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.020091+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:30.020091+0000","last_peered":"2021-01-15T18:46:30.020091+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:30.020091+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6540,"ondisk_log_size":6540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26640384,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"47'6623","reported_seq":"7222","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.985187+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:29.985187+0000","last_peered":"2021-01-15T18:46:29.985187+0000","last_clean":"2021-01-15T18:46:29.985187+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:29.985187+0000","last_undegraded":"2021-01-15T18:46:29.985187+0000","last_fullsized":"2021-01-15T18:46:29.985187+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6623,"ondisk_log_size":6623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26255526,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6621,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"47'6320","reported_seq":"6538","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.933691+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:29.933691+0000","last_peered":"2021-01-15T18:46:29.933691+0000","last_clean":"2021-01-15T18:46:29.933691+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:29.933691+0000","last_undegraded":"2021-01-15T18:46:29.933691+0000","last_fullsized":"2021-01-15T18:46:29.933691+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6320,"ondisk_log_size":6320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26263718,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6318,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"47'6469","reported_seq":"6773","reported_epoch":"47","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.059796+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:30.059796+0000","last_peered":"2021-01-15T18:46:30.059796+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:30.059796+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6469,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26673318,"num_objects":3257,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"46'6596","reported_seq":"6758","reported_epoch":"46","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:28.521195+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:28.521195+0000","last_peered":"2021-01-15T18:46:28.521195+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:28.521195+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6596,"ondisk_log_size":6596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26710099,"num_objects":3261,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6595,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"47'6530","reported_seq":"9472","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.644197+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:29.644197+0000","last_peered":"2021-01-15T18:46:29.644197+0000","last_clean":"2021-01-15T18:46:29.644197+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:29.644197+0000","last_undegraded":"2021-01-15T18:46:29.644197+0000","last_fullsized":"2021-01-15T18:46:29.644197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6530,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26202195,"num_objects":3199,"num_object_clones":0,"num_object_copies":12796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6529,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'6549","reported_seq":"9497","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.999230+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:29.999230+0000","last_peered":"2021-01-15T18:46:29.999230+0000","last_clean":"2021-01-15T18:46:29.999230+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:29.999230+0000","last_undegraded":"2021-01-15T18:46:29.999230+0000","last_fullsized":"2021-01-15T18:46:29.999230+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6549,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26284281,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"47'6367","reported_seq":"6614","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:29.804105+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:29.804105+0000","last_peered":"2021-01-15T18:46:29.804105+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:29.804105+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6367,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26108070,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"47'6371","reported_seq":"9206","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:30.033784+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:30.033784+0000","last_peered":"2021-01-15T18:46:30.033784+0000","last_clean":"2021-01-15T18:46:30.033784+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:30.033784+0000","last_undegraded":"2021-01-15T18:46:30.033784+0000","last_fullsized":"2021-01-15T18:46:30.033784+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6371,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26181798,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"46'6349","reported_seq":"9250","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:29.358982+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:29.358982+0000","last_peered":"2021-01-15T18:46:29.358982+0000","last_clean":"2021-01-15T18:46:29.358982+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:29.358982+0000","last_undegraded":"2021-01-15T18:46:29.358982+0000","last_fullsized":"2021-01-15T18:46:29.358982+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6349,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25608358,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'6630","reported_seq":"6863","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.912236+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:29.912236+0000","last_peered":"2021-01-15T18:46:29.912236+0000","last_clean":"2021-01-15T18:46:29.912236+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:29.912236+0000","last_undegraded":"2021-01-15T18:46:29.912236+0000","last_fullsized":"2021-01-15T18:46:29.912236+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6630,"ondisk_log_size":6630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26673318,"num_objects":3257,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'6761","reported_seq":"7034","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.021639+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:30.021639+0000","last_peered":"2021-01-15T18:46:30.021639+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:30.021639+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6761,"ondisk_log_size":6761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26878284,"num_objects":3283,"num_object_clones":0,"num_object_copies":13132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"46'6347","reported_seq":"9218","reported_epoch":"46","state":"active+clean","last_fresh":"2021-01-15T18:46:28.384230+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:28.384230+0000","last_peered":"2021-01-15T18:46:28.384230+0000","last_clean":"2021-01-15T18:46:28.384230+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:28.384230+0000","last_undegraded":"2021-01-15T18:46:28.384230+0000","last_fullsized":"2021-01-15T18:46:28.384230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6347,"ondisk_log_size":6347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26149030,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6345,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":419940652,"num_objects":51277,"num_object_clones":0,"num_object_copies":205108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103448,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13450,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":840671232,"data_stored":840671232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103475,"ondisk_log_size":103475,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409648,"kb_used_data":205648,"kb_used_omap":0,"kb_used_meta":203967,"kb_avail":93962192,"statfs":{"total":96636764160,"available":96217284608,"internally_reserved":0,"allocated":210583552,"data_stored":210340235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208862260},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,10,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":6.585},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":6.7510000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.304}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":410628,"kb_used_data":207192,"kb_used_omap":0,"kb_used_meta":203392,"kb_avail":93961212,"statfs":{"total":96636764160,"available":96216281088,"internally_reserved":0,"allocated":212164608,"data_stored":211921291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":208273408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.873}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.1839999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401648,"kb_used_data":205824,"kb_used_omap":0,"kb_used_meta":195775,"kb_avail":93970192,"statfs":{"total":96636764160,"available":96225476608,"internally_reserved":0,"allocated":210763776,"data_stored":210520459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":200473652},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":2.5379999999999998},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":6.1959999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":1.0820000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":1.131}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":1.7070000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398124,"kb_used_data":205976,"kb_used_omap":0,"kb_used_meta":192127,"kb_avail":93973716,"statfs":{"total":96636764160,"available":96229085184,"internally_reserved":0,"allocated":210919424,"data_stored":210676107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":196738100},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":10.789999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":3.7650000000000001},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":3.6539999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.80500000000000005}]}]}]}} 2021-01-15T18:46:34.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:34.899 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:34.900 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7955/204116 objects degraded (3.897%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":445,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51029,"data_bytes":417909036,"bytes_used":1654480896,"bytes_avail":384892575744,"bytes_total":386547056640,"degraded_objects":7955,"degraded_total":204116,"degraded_ratio":0.038972936957416368,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:34.956 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:34.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:36.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:36.206 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:36.237 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2021-01-15T18:46:34.561577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417909036,"num_objects":51029,"num_object_clones":0,"num_object_copies":204116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103696,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13452,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103723,"ondisk_log_size":103723,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1615704,"kb_used_data":816240,"kb_used_omap":2,"kb_used_meta":799293,"kb_avail":375871656,"statfs":{"total":386547056640,"available":384892575744,"internally_reserved":0,"allocated":835829760,"data_stored":834836770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":818476188},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,9,19,17,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":111,"apply_latency_ms":111,"commit_latency_ns":111000000,"apply_latency_ns":111000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11689984,"num_objects":-1427,"num_object_clones":0,"num_object_copies":-5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002453"},"pg_stats":[{"pgid":"1.f","version":"47'6311","reported_seq":"6576","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:29.865872+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:29.865872+0000","last_peered":"2021-01-15T18:46:29.865872+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:29.865872+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6311,"ondisk_log_size":6311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26189824,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1947}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"47'6360","reported_seq":"9290","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:30.035697+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:30.035697+0000","last_peered":"2021-01-15T18:46:30.035697+0000","last_clean":"2021-01-15T18:46:30.035697+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:30.035697+0000","last_undegraded":"2021-01-15T18:46:30.035697+0000","last_fullsized":"2021-01-15T18:46:30.035697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6360,"ondisk_log_size":6360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25403558,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6428","reported_seq":"9367","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:33.358227+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:33.358227+0000","last_peered":"2021-01-15T18:46:33.358227+0000","last_clean":"2021-01-15T18:46:33.358227+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:33.358227+0000","last_undegraded":"2021-01-15T18:46:33.358227+0000","last_fullsized":"2021-01-15T18:46:33.358227+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25096299,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6427,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"47'6540","reported_seq":"6793","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.020091+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:30.020091+0000","last_peered":"2021-01-15T18:46:30.020091+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:30.020091+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6540,"ondisk_log_size":6540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26640384,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1918}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"47'6623","reported_seq":"7222","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.985187+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:29.985187+0000","last_peered":"2021-01-15T18:46:29.985187+0000","last_clean":"2021-01-15T18:46:29.985187+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:29.985187+0000","last_undegraded":"2021-01-15T18:46:29.985187+0000","last_fullsized":"2021-01-15T18:46:29.985187+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6623,"ondisk_log_size":6623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26255526,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6621,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"47'6320","reported_seq":"6538","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.933691+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:29.933691+0000","last_peered":"2021-01-15T18:46:29.933691+0000","last_clean":"2021-01-15T18:46:29.933691+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:29.933691+0000","last_undegraded":"2021-01-15T18:46:29.933691+0000","last_fullsized":"2021-01-15T18:46:29.933691+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6320,"ondisk_log_size":6320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26263718,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6318,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"47'6469","reported_seq":"6773","reported_epoch":"47","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.059796+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:30.059796+0000","last_peered":"2021-01-15T18:46:30.059796+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:30.059796+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6469,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26673318,"num_objects":3257,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1982}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6651","reported_seq":"6815","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:33.559294+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:33.559294+0000","last_peered":"2021-01-15T18:46:33.559294+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:33.559294+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6651,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26259539,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"47'6530","reported_seq":"9472","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.644197+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:29.644197+0000","last_peered":"2021-01-15T18:46:29.644197+0000","last_clean":"2021-01-15T18:46:29.644197+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:29.644197+0000","last_undegraded":"2021-01-15T18:46:29.644197+0000","last_fullsized":"2021-01-15T18:46:29.644197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6530,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26202195,"num_objects":3199,"num_object_clones":0,"num_object_copies":12796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6529,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'6549","reported_seq":"9497","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.999230+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:29.999230+0000","last_peered":"2021-01-15T18:46:29.999230+0000","last_clean":"2021-01-15T18:46:29.999230+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:29.999230+0000","last_undegraded":"2021-01-15T18:46:29.999230+0000","last_fullsized":"2021-01-15T18:46:29.999230+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6549,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26284281,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"47'6367","reported_seq":"6614","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:29.804105+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:29.804105+0000","last_peered":"2021-01-15T18:46:29.804105+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:29.804105+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6367,"ondisk_log_size":6367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26108070,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1896}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"47'6371","reported_seq":"9206","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:30.033784+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:30.033784+0000","last_peered":"2021-01-15T18:46:30.033784+0000","last_clean":"2021-01-15T18:46:30.033784+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:30.033784+0000","last_undegraded":"2021-01-15T18:46:30.033784+0000","last_fullsized":"2021-01-15T18:46:30.033784+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6371,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26181798,"num_objects":3197,"num_object_clones":0,"num_object_copies":12788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6408","reported_seq":"9311","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.188533+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:34.188533+0000","last_peered":"2021-01-15T18:46:34.188533+0000","last_clean":"2021-01-15T18:46:34.188533+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:34.188533+0000","last_undegraded":"2021-01-15T18:46:34.188533+0000","last_fullsized":"2021-01-15T18:46:34.188533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6408,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25125030,"num_objects":3068,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'6630","reported_seq":"6863","reported_epoch":"47","state":"active+clean","last_fresh":"2021-01-15T18:46:29.912236+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:29.912236+0000","last_peered":"2021-01-15T18:46:29.912236+0000","last_clean":"2021-01-15T18:46:29.912236+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:29.912236+0000","last_undegraded":"2021-01-15T18:46:29.912236+0000","last_fullsized":"2021-01-15T18:46:29.912236+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6630,"ondisk_log_size":6630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26673318,"num_objects":3257,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6628,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"47'6761","reported_seq":"7034","reported_epoch":"47","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:30.021639+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:30.021639+0000","last_peered":"2021-01-15T18:46:30.021639+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:30.021639+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6761,"ondisk_log_size":6761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26878284,"num_objects":3283,"num_object_clones":0,"num_object_copies":13132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6405","reported_seq":"9278","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:33.578655+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:33.578655+0000","last_peered":"2021-01-15T18:46:33.578655+0000","last_clean":"2021-01-15T18:46:33.578655+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:33.578655+0000","last_undegraded":"2021-01-15T18:46:33.578655+0000","last_fullsized":"2021-01-15T18:46:33.578655+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6405,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25673894,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417909036,"num_objects":51029,"num_object_clones":0,"num_object_copies":204116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103696,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13452,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":832147456,"data_stored":832147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103723,"ondisk_log_size":103723,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215188,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409648,"kb_used_data":205648,"kb_used_omap":0,"kb_used_meta":203967,"kb_avail":93962192,"statfs":{"total":96636764160,"available":96217284608,"internally_reserved":0,"allocated":210583552,"data_stored":210340235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208862260},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,10,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":6.585},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":6.7510000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.304}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400260,"kb_used_data":202960,"kb_used_omap":0,"kb_used_meta":197248,"kb_avail":93971580,"statfs":{"total":96636764160,"available":96226897920,"internally_reserved":0,"allocated":207831040,"data_stored":207577862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":201981952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.873}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.91800000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":8.9670000000000005},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.988}]}]},{"osd":0,"up_from":22,"seq":94489280601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407672,"kb_used_data":201656,"kb_used_omap":0,"kb_used_meta":205951,"kb_avail":93964168,"statfs":{"total":96636764160,"available":96219308032,"internally_reserved":0,"allocated":206495744,"data_stored":206242566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":210893876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.0589999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.73099999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.3650000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":7.569}]}]},{"osd":1,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398124,"kb_used_data":205976,"kb_used_omap":0,"kb_used_meta":192127,"kb_avail":93973716,"statfs":{"total":96636764160,"available":96229085184,"internally_reserved":0,"allocated":210919424,"data_stored":210676107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":196738100},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":10.789999999999999},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":3.7650000000000001},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":3.6539999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.80500000000000005}]}]}]}} 2021-01-15T18:46:39.240 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:40.339 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:40.392 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2021-01-15T18:46:38.562506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411535660,"num_objects":50251,"num_object_clones":0,"num_object_copies":201004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104474,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13528,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104501,"ondisk_log_size":104501,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1598828,"kb_used_data":807276,"kb_used_omap":2,"kb_used_meta":791357,"kb_avail":375888532,"statfs":{"total":386547056640,"available":384909856768,"internally_reserved":0,"allocated":826650624,"data_stored":825637912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":810349724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,16,14,22,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":107,"apply_latency_ms":107,"commit_latency_ns":107000000,"apply_latency_ns":107000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15581184,"num_objects":-1902,"num_object_clones":0,"num_object_copies":-7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1902,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.950522"},"pg_stats":[{"pgid":"1.f","version":"48'6374","reported_seq":"6640","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:34.878352+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:34.878352+0000","last_peered":"2021-01-15T18:46:34.878352+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:34.878352+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6374,"ondisk_log_size":6374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25673728,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6424","reported_seq":"9355","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:35.126799+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:35.126799+0000","last_peered":"2021-01-15T18:46:35.126799+0000","last_clean":"2021-01-15T18:46:35.126799+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:35.126799+0000","last_undegraded":"2021-01-15T18:46:35.126799+0000","last_fullsized":"2021-01-15T18:46:35.126799+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6424,"ondisk_log_size":6424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24879270,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6428","reported_seq":"9367","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:33.358227+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:33.358227+0000","last_peered":"2021-01-15T18:46:33.358227+0000","last_clean":"2021-01-15T18:46:33.358227+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:33.358227+0000","last_undegraded":"2021-01-15T18:46:33.358227+0000","last_fullsized":"2021-01-15T18:46:33.358227+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6428,"ondisk_log_size":6428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25096299,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6427,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6605","reported_seq":"6859","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:35.018020+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:35.018020+0000","last_peered":"2021-01-15T18:46:35.018020+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:35.018020+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6605,"ondisk_log_size":6605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26107904,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6605,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6694","reported_seq":"7293","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:35.125514+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:35.125514+0000","last_peered":"2021-01-15T18:46:35.125514+0000","last_clean":"2021-01-15T18:46:35.125514+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:35.125514+0000","last_undegraded":"2021-01-15T18:46:35.125514+0000","last_fullsized":"2021-01-15T18:46:35.125514+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6694,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25673894,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6383","reported_seq":"6602","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.891685+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:34.891685+0000","last_peered":"2021-01-15T18:46:34.891685+0000","last_clean":"2021-01-15T18:46:34.891685+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:34.891685+0000","last_undegraded":"2021-01-15T18:46:34.891685+0000","last_fullsized":"2021-01-15T18:46:34.891685+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6383,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25747622,"num_objects":3144,"num_object_clones":0,"num_object_copies":12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6533","reported_seq":"6990","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:34.963923+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:34.963923+0000","last_peered":"2021-01-15T18:46:34.963923+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:34.963923+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26149030,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6651","reported_seq":"6815","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:33.559294+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:33.559294+0000","last_peered":"2021-01-15T18:46:33.559294+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:33.559294+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6651,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26259539,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6602","reported_seq":"9545","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.959023+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:34.959023+0000","last_peered":"2021-01-15T18:46:34.959023+0000","last_clean":"2021-01-15T18:46:34.959023+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:34.959023+0000","last_undegraded":"2021-01-15T18:46:34.959023+0000","last_fullsized":"2021-01-15T18:46:34.959023+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6602,"ondisk_log_size":6602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25612371,"num_objects":3127,"num_object_clones":0,"num_object_copies":12508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6601,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6619","reported_seq":"9568","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.828808+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:34.828808+0000","last_peered":"2021-01-15T18:46:34.828808+0000","last_clean":"2021-01-15T18:46:34.828808+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:34.828808+0000","last_undegraded":"2021-01-15T18:46:34.828808+0000","last_fullsized":"2021-01-15T18:46:34.828808+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6619,"ondisk_log_size":6619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25710841,"num_objects":3140,"num_object_clones":0,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6421","reported_seq":"6668","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:34.792249+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:34.792249+0000","last_peered":"2021-01-15T18:46:34.792249+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:34.792249+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6421,"ondisk_log_size":6421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25665702,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6434","reported_seq":"9271","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.959597+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:34.959597+0000","last_peered":"2021-01-15T18:46:34.959597+0000","last_clean":"2021-01-15T18:46:34.959597+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:34.959597+0000","last_undegraded":"2021-01-15T18:46:34.959597+0000","last_fullsized":"2021-01-15T18:46:34.959597+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6434,"ondisk_log_size":6434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25665702,"num_objects":3134,"num_object_clones":0,"num_object_copies":12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6432,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6408","reported_seq":"9311","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:34.188533+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:34.188533+0000","last_peered":"2021-01-15T18:46:34.188533+0000","last_clean":"2021-01-15T18:46:34.188533+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:34.188533+0000","last_undegraded":"2021-01-15T18:46:34.188533+0000","last_fullsized":"2021-01-15T18:46:34.188533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6408,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25125030,"num_objects":3068,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'6701","reported_seq":"6935","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:35.125190+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:35.125190+0000","last_peered":"2021-01-15T18:46:35.125190+0000","last_clean":"2021-01-15T18:46:35.125190+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:35.125190+0000","last_undegraded":"2021-01-15T18:46:35.125190+0000","last_fullsized":"2021-01-15T18:46:35.125190+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6701,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26091686,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'6819","reported_seq":"7093","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:34.872301+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:34.872301+0000","last_peered":"2021-01-15T18:46:34.872301+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:34.872301+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6819,"ondisk_log_size":6819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26403148,"num_objects":3225,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6405","reported_seq":"9278","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:33.578655+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:33.578655+0000","last_peered":"2021-01-15T18:46:33.578655+0000","last_clean":"2021-01-15T18:46:33.578655+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:33.578655+0000","last_undegraded":"2021-01-15T18:46:33.578655+0000","last_fullsized":"2021-01-15T18:46:33.578655+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6405,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25673894,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6403,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":411535660,"num_objects":50251,"num_object_clones":0,"num_object_copies":201004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7879,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104474,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13528,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823574528,"data_stored":823574528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104501,"ondisk_log_size":104501,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215189,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399320,"kb_used_data":201588,"kb_used_omap":0,"kb_used_meta":197695,"kb_avail":93972520,"statfs":{"total":96636764160,"available":96227860480,"internally_reserved":0,"allocated":206426112,"data_stored":206172934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":202439732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":4.5949999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.60499999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:45:37 2021","interfaces":[{"interface":"back","average":{"1min":5.2270000000000003,"5min":4.5279999999999996,"15min":3.5569999999999999},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":26.533999999999999,"5min":30.129000000000001,"15min":30.129000000000001},"last":4.3600000000000003},{"interface":"front","average":{"1min":6.0259999999999998,"5min":4.133,"15min":2.6709999999999998},"min":{"1min":0.78000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":26.635000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":4.476}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":6.2699999999999996},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":5.101}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400260,"kb_used_data":202960,"kb_used_omap":0,"kb_used_meta":197248,"kb_avail":93971580,"statfs":{"total":96636764160,"available":96226897920,"internally_reserved":0,"allocated":207831040,"data_stored":207577862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":201981952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.873}]},{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":3.02,"5min":2.9409999999999998,"15min":2.504},"min":{"1min":0.443,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":15.609,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.992,"15min":2.0430000000000001},"min":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.0119999999999996,"5min":20.98,"15min":20.98},"last":0.91800000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":8.9670000000000005},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.988}]}]},{"osd":0,"up_from":22,"seq":94489280601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407672,"kb_used_data":201656,"kb_used_omap":0,"kb_used_meta":205951,"kb_avail":93964168,"statfs":{"total":96636764160,"available":96219308032,"internally_reserved":0,"allocated":206495744,"data_stored":206242566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":210893876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:45:34 2021","interfaces":[{"interface":"back","average":{"1min":6.125,"5min":3.246,"15min":3.0409999999999999},"min":{"1min":0.442,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":4.5,"5min":4.2249999999999996,"15min":2.8029999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.222999999999999,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.0589999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.73099999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.3650000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":7.569}]}]},{"osd":1,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391576,"kb_used_data":201072,"kb_used_omap":0,"kb_used_meta":190463,"kb_avail":93980264,"statfs":{"total":96636764160,"available":96235790336,"internally_reserved":0,"allocated":205897728,"data_stored":205644550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":195034164},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":2.161},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.69599999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":1.4259999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.94399999999999995}]}]}]}} 2021-01-15T18:46:40.394 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:41.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:41.794 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7833/196236 objects degraded (3.992%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":452,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49059,"data_bytes":401770796,"bytes_used":1639407616,"bytes_avail":384907649024,"bytes_total":386547056640,"degraded_objects":7833,"degraded_total":196236,"degraded_ratio":0.039916223322937684,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:41.807 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:41.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:43.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:43.282 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2021-01-15T18:46:42.569542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":401770796,"num_objects":49059,"num_object_clones":0,"num_object_copies":196236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105666,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13574,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105693,"ondisk_log_size":105693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1600984,"kb_used_data":788428,"kb_used_omap":2,"kb_used_meta":812349,"kb_avail":375886376,"statfs":{"total":386547056640,"available":384907649024,"internally_reserved":0,"allocated":807350272,"data_stored":806337560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":831845532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,4,14,15,25,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":119,"apply_latency_ms":119,"commit_latency_ns":119000000,"apply_latency_ns":119000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18169856,"num_objects":-2218,"num_object_clones":0,"num_object_copies":-8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.945337"},"pg_stats":[{"pgid":"1.f","version":"48'6453","reported_seq":"6719","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:39.913947+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:39.913947+0000","last_peered":"2021-01-15T18:46:39.913947+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:39.913947+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6453,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25026560,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6490","reported_seq":"9422","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:39.909548+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:39.909548+0000","last_peered":"2021-01-15T18:46:39.909548+0000","last_clean":"2021-01-15T18:46:39.909548+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:39.909548+0000","last_undegraded":"2021-01-15T18:46:39.909548+0000","last_fullsized":"2021-01-15T18:46:39.909548+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6490,"ondisk_log_size":6490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24338598,"num_objects":2972,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6515","reported_seq":"9454","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:38.582662+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:38.582662+0000","last_peered":"2021-01-15T18:46:38.582662+0000","last_clean":"2021-01-15T18:46:38.582662+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:38.582662+0000","last_undegraded":"2021-01-15T18:46:38.582662+0000","last_fullsized":"2021-01-15T18:46:38.582662+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6515,"ondisk_log_size":6515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24383595,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6514,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6679","reported_seq":"6933","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:39.915132+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:39.915132+0000","last_peered":"2021-01-15T18:46:39.915132+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:39.915132+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6679,"ondisk_log_size":6679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25501696,"num_objects":3113,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6679,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6770","reported_seq":"7369","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:40.058784+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:40.058784+0000","last_peered":"2021-01-15T18:46:40.058784+0000","last_clean":"2021-01-15T18:46:40.058784+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:40.058784+0000","last_undegraded":"2021-01-15T18:46:40.058784+0000","last_fullsized":"2021-01-15T18:46:40.058784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6770,"ondisk_log_size":6770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25051302,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6464","reported_seq":"6682","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:40.135045+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:40.135045+0000","last_peered":"2021-01-15T18:46:40.135045+0000","last_clean":"2021-01-15T18:46:40.135045+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:40.135045+0000","last_undegraded":"2021-01-15T18:46:40.135045+0000","last_fullsized":"2021-01-15T18:46:40.135045+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6464,"ondisk_log_size":6464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25084070,"num_objects":3063,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6462,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6607","reported_seq":"7156","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:39.952874+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:39.952874+0000","last_peered":"2021-01-15T18:46:39.952874+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:39.952874+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6607,"ondisk_log_size":6607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25542822,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6605,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1966}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6734","reported_seq":"6898","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:38.578950+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:38.578950+0000","last_peered":"2021-01-15T18:46:38.578950+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:38.578950+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6734,"ondisk_log_size":6734,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25579603,"num_objects":3123,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6672","reported_seq":"9615","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:39.989708+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:39.989708+0000","last_peered":"2021-01-15T18:46:39.989708+0000","last_clean":"2021-01-15T18:46:39.989708+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:39.989708+0000","last_undegraded":"2021-01-15T18:46:39.989708+0000","last_fullsized":"2021-01-15T18:46:39.989708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6672,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25038931,"num_objects":3057,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6695","reported_seq":"9643","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:39.904782+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:39.904782+0000","last_peered":"2021-01-15T18:46:39.904782+0000","last_clean":"2021-01-15T18:46:39.904782+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:39.904782+0000","last_undegraded":"2021-01-15T18:46:39.904782+0000","last_fullsized":"2021-01-15T18:46:39.904782+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6695,"ondisk_log_size":6695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25088249,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6491","reported_seq":"6738","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:40.015624+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:40.015624+0000","last_peered":"2021-01-15T18:46:40.015624+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:40.015624+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6491,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25092262,"num_objects":3064,"num_object_clones":0,"num_object_copies":12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6512","reported_seq":"9349","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:40.066078+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:40.066078+0000","last_peered":"2021-01-15T18:46:40.066078+0000","last_clean":"2021-01-15T18:46:40.066078+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:40.066078+0000","last_undegraded":"2021-01-15T18:46:40.066078+0000","last_fullsized":"2021-01-15T18:46:40.066078+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6512,"ondisk_log_size":6512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25026726,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6510,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6477","reported_seq":"9380","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:39.437477+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:39.437477+0000","last_peered":"2021-01-15T18:46:39.437477+0000","last_clean":"2021-01-15T18:46:39.437477+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:39.437477+0000","last_undegraded":"2021-01-15T18:46:39.437477+0000","last_fullsized":"2021-01-15T18:46:39.437477+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6477,"ondisk_log_size":6477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24559782,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6475,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'6777","reported_seq":"7010","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:40.050609+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:40.050609+0000","last_peered":"2021-01-15T18:46:40.050609+0000","last_clean":"2021-01-15T18:46:40.050609+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:40.050609+0000","last_undegraded":"2021-01-15T18:46:40.050609+0000","last_fullsized":"2021-01-15T18:46:40.050609+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6777,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25469094,"num_objects":3110,"num_object_clones":0,"num_object_copies":12440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'6875","reported_seq":"7149","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:39.916496+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:39.916496+0000","last_peered":"2021-01-15T18:46:39.916496+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:39.916496+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6875,"ondisk_log_size":6875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25944396,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1784}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6482","reported_seq":"9355","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:38.519247+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:38.519247+0000","last_peered":"2021-01-15T18:46:38.519247+0000","last_clean":"2021-01-15T18:46:38.519247+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:38.519247+0000","last_undegraded":"2021-01-15T18:46:38.519247+0000","last_fullsized":"2021-01-15T18:46:38.519247+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6482,"ondisk_log_size":6482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25043110,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":401770796,"num_objects":49059,"num_object_clones":0,"num_object_copies":196236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105666,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13574,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":804294656,"data_stored":804294656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105693,"ondisk_log_size":105693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215190,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":404896,"kb_used_data":196692,"kb_used_omap":0,"kb_used_meta":208127,"kb_avail":93966944,"statfs":{"total":96636764160,"available":96222150656,"internally_reserved":0,"allocated":201412608,"data_stored":201159430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":213122100},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.1499999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.244},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.331}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":2.823},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.6259999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405980,"kb_used_data":198400,"kb_used_omap":0,"kb_used_meta":207552,"kb_avail":93965860,"statfs":{"total":96636764160,"available":96221040640,"internally_reserved":0,"allocated":203161600,"data_stored":202908422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":212533248},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":3.8079999999999998},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":4.6399999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":4.7240000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":3.6669999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":4.7519999999999998},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":3.911}]}]},{"osd":0,"up_from":22,"seq":94489280602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396952,"kb_used_data":197060,"kb_used_omap":0,"kb_used_meta":199871,"kb_avail":93974888,"statfs":{"total":96636764160,"available":96230285312,"internally_reserved":0,"allocated":201789440,"data_stored":201536262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":204667956},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,3,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":6.9480000000000004},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":3.4359999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":10.204000000000001},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":10.225}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":21.138000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393156,"kb_used_data":196276,"kb_used_omap":0,"kb_used_meta":196799,"kb_avail":93978684,"statfs":{"total":96636764160,"available":96234172416,"internally_reserved":0,"allocated":200986624,"data_stored":200733446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":201522228},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,3,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":4.383,"15min":3.2589999999999999},"min":{"1min":0.47699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":8.7579999999999991,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.173,"5min":2.5049999999999999,"15min":2.069},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.7400000000000002,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.64300000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":2.7389999999999999,"15min":2.7290000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.2859999999999996,"5min":21.791,"15min":21.791},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.371,"5min":2.097,"15min":1.8009999999999999},"min":{"1min":0.65700000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.5229999999999997,"5min":19.381,"15min":19.381},"last":1.651}]},{"osd":3,"last update":"Fri Jan 15 18:45:41 2021","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":3.4849999999999999,"15min":3.956},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":10.281000000000001,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.927},{"interface":"front","average":{"1min":2.8730000000000002,"5min":2.5510000000000002,"15min":2.605},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":9.7089999999999996,"5min":13.478999999999999,"15min":13.478999999999999},"last":2.0270000000000001}]}]}]}} 2021-01-15T18:46:46.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:48.584 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:48.611 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2021-01-15T18:46:46.574484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392997164,"num_objects":47988,"num_object_clones":0,"num_object_copies":191952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47988,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106737,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13629,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106764,"ondisk_log_size":106764,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1575564,"kb_used_data":771840,"kb_used_omap":2,"kb_used_meta":803581,"kb_avail":375911796,"statfs":{"total":386547056640,"available":384933679104,"internally_reserved":0,"allocated":790364160,"data_stored":789351448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":822867100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,8,5,14,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":116,"apply_latency_ms":116,"commit_latency_ns":116000000,"apply_latency_ns":116000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24911872,"num_objects":-3041,"num_object_clones":0,"num_object_copies":-12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012907"},"pg_stats":[{"pgid":"1.f","version":"48'6513","reported_seq":"6778","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:44.895604+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:44.895604+0000","last_peered":"2021-01-15T18:46:44.895604+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:44.895604+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6513,"ondisk_log_size":6513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24535040,"num_objects":2995,"num_object_clones":0,"num_object_copies":11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6513,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6548","reported_seq":"9480","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.042028+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:45.042028+0000","last_peered":"2021-01-15T18:46:45.042028+0000","last_clean":"2021-01-15T18:46:45.042028+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:45.042028+0000","last_undegraded":"2021-01-15T18:46:45.042028+0000","last_fullsized":"2021-01-15T18:46:45.042028+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6548,"ondisk_log_size":6548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23863462,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6546,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6580","reported_seq":"9519","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:43.487350+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:43.487350+0000","last_peered":"2021-01-15T18:46:43.487350+0000","last_clean":"2021-01-15T18:46:43.487350+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:43.487350+0000","last_undegraded":"2021-01-15T18:46:43.487350+0000","last_fullsized":"2021-01-15T18:46:43.487350+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6580,"ondisk_log_size":6580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23851115,"num_objects":2913,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6579,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6750","reported_seq":"7004","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:45.023433+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:45.023433+0000","last_peered":"2021-01-15T18:46:45.023433+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:45.023433+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6750,"ondisk_log_size":6750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24920064,"num_objects":3042,"num_object_clones":0,"num_object_copies":12168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6750,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6849","reported_seq":"7448","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.056860+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:45.056860+0000","last_peered":"2021-01-15T18:46:45.056860+0000","last_clean":"2021-01-15T18:46:45.056860+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:45.056860+0000","last_undegraded":"2021-01-15T18:46:45.056860+0000","last_fullsized":"2021-01-15T18:46:45.056860+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6849,"ondisk_log_size":6849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24404134,"num_objects":2980,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6521","reported_seq":"6739","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.138918+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:45.138918+0000","last_peered":"2021-01-15T18:46:45.138918+0000","last_clean":"2021-01-15T18:46:45.138918+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:45.138918+0000","last_undegraded":"2021-01-15T18:46:45.138918+0000","last_fullsized":"2021-01-15T18:46:45.138918+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6521,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24617126,"num_objects":3006,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6680","reported_seq":"7339","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:45.049114+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:45.049114+0000","last_peered":"2021-01-15T18:46:45.049114+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:45.049114+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6680,"ondisk_log_size":6680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24944806,"num_objects":3046,"num_object_clones":0,"num_object_copies":12184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6804","reported_seq":"6965","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:43.549330+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:43.549330+0000","last_peered":"2021-01-15T18:46:43.549330+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:43.549330+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6804,"ondisk_log_size":6804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25006163,"num_objects":3053,"num_object_clones":0,"num_object_copies":12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6803,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1640}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6724","reported_seq":"9666","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.068797+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:45.068797+0000","last_peered":"2021-01-15T18:46:45.068797+0000","last_clean":"2021-01-15T18:46:45.068797+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:45.068797+0000","last_undegraded":"2021-01-15T18:46:45.068797+0000","last_fullsized":"2021-01-15T18:46:45.068797+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24612947,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6760","reported_seq":"9708","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.041944+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:45.041944+0000","last_peered":"2021-01-15T18:46:45.041944+0000","last_clean":"2021-01-15T18:46:45.041944+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:45.041944+0000","last_undegraded":"2021-01-15T18:46:45.041944+0000","last_fullsized":"2021-01-15T18:46:45.041944+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6760,"ondisk_log_size":6760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24555769,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6757,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6549","reported_seq":"6797","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:44.920488+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:44.920488+0000","last_peered":"2021-01-15T18:46:44.920488+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:44.920488+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6549,"ondisk_log_size":6549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24617126,"num_objects":3006,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6547,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6579","reported_seq":"9416","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:44.983138+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:44.983138+0000","last_peered":"2021-01-15T18:46:44.983138+0000","last_clean":"2021-01-15T18:46:44.983138+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:44.983138+0000","last_undegraded":"2021-01-15T18:46:44.983138+0000","last_fullsized":"2021-01-15T18:46:44.983138+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6579,"ondisk_log_size":6579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24477862,"num_objects":2989,"num_object_clones":0,"num_object_copies":11956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6577,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6540","reported_seq":"9443","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:44.117694+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:44.117694+0000","last_peered":"2021-01-15T18:46:44.117694+0000","last_clean":"2021-01-15T18:46:44.117694+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:44.117694+0000","last_undegraded":"2021-01-15T18:46:44.117694+0000","last_fullsized":"2021-01-15T18:46:44.117694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6540,"ondisk_log_size":6540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24043686,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'6857","reported_seq":"7091","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:45.046687+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:45.046687+0000","last_peered":"2021-01-15T18:46:45.046687+0000","last_clean":"2021-01-15T18:46:45.046687+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:45.046687+0000","last_undegraded":"2021-01-15T18:46:45.046687+0000","last_fullsized":"2021-01-15T18:46:45.046687+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6857,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24813734,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'6957","reported_seq":"7231","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:45.061933+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:45.061933+0000","last_peered":"2021-01-15T18:46:45.061933+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:45.061933+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6957,"ondisk_log_size":6957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25272652,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6954,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6553","reported_seq":"9426","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:43.439941+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:43.439941+0000","last_peered":"2021-01-15T18:46:43.439941+0000","last_clean":"2021-01-15T18:46:43.439941+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:43.439941+0000","last_undegraded":"2021-01-15T18:46:43.439941+0000","last_fullsized":"2021-01-15T18:46:43.439941+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6553,"ondisk_log_size":6553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24461478,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":392997164,"num_objects":47988,"num_object_clones":0,"num_object_copies":191952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47988,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106737,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13629,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786595840,"data_stored":786595840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106764,"ondisk_log_size":106764,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215191,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394400,"kb_used_data":192528,"kb_used_omap":0,"kb_used_meta":201855,"kb_avail":93977440,"statfs":{"total":96636764160,"available":96232898560,"internally_reserved":0,"allocated":197148672,"data_stored":196895494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":206699572},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,8,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.94699999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.179},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.274}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":2.0150000000000001},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.0600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395272,"kb_used_data":193820,"kb_used_omap":0,"kb_used_meta":201408,"kb_avail":93976568,"statfs":{"total":96636764160,"available":96232005632,"internally_reserved":0,"allocated":198471680,"data_stored":198218502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":206241792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.77500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.139},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":5.2930000000000001},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":6.5039999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402748,"kb_used_data":192664,"kb_used_omap":0,"kb_used_meta":210047,"kb_avail":93969092,"statfs":{"total":96636764160,"available":96224350208,"internally_reserved":0,"allocated":197287936,"data_stored":197034758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":215088180},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":1.0589999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.77600000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.83999999999999997},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":4.5060000000000002},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":24.734999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383144,"kb_used_data":192828,"kb_used_omap":0,"kb_used_meta":190271,"kb_avail":93988696,"statfs":{"total":96636764160,"available":96244424704,"internally_reserved":0,"allocated":197455872,"data_stored":197202694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":194837556},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":14.013},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.145}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.901},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":2.0049999999999999}]}]}]}} 2021-01-15T18:46:48.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:50.020 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:50.021 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7778/191952 objects degraded (4.052%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47988,"data_bytes":392997164,"bytes_used":1613377536,"bytes_avail":384933679104,"bytes_total":386547056640,"degraded_objects":7778,"degraded_total":191952,"degraded_ratio":0.040520546803367502,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:50.084 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:50.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:50.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:50.787 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:50.837 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2021-01-15T18:46:50.575660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":383887660,"num_objects":46876,"num_object_clones":0,"num_object_copies":187504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107849,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107876,"ondisk_log_size":107876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1582472,"kb_used_data":754128,"kb_used_omap":2,"kb_used_meta":828093,"kb_avail":375904888,"statfs":{"total":386547056640,"available":384926605312,"internally_reserved":0,"allocated":772227072,"data_stored":771214360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":847967388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,5,5,9,11,9,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":116,"apply_latency_ms":116,"commit_latency_ns":116000000,"apply_latency_ns":116000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27648000,"num_objects":-3375,"num_object_clones":0,"num_object_copies":-13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013154"},"pg_stats":[{"pgid":"1.f","version":"48'6581","reported_seq":"6847","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.072243+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:50.072243+0000","last_peered":"2021-01-15T18:46:50.072243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:50.072243+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6581,"ondisk_log_size":6581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6581,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6615","reported_seq":"9547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.989178+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:49.989178+0000","last_peered":"2021-01-15T18:46:49.989178+0000","last_clean":"2021-01-15T18:46:49.989178+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:49.989178+0000","last_undegraded":"2021-01-15T18:46:49.989178+0000","last_fullsized":"2021-01-15T18:46:49.989178+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6615,"ondisk_log_size":6615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23314598,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6613,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6645","reported_seq":"9584","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:48.321986+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:48.321986+0000","last_peered":"2021-01-15T18:46:48.321986+0000","last_clean":"2021-01-15T18:46:48.321986+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:48.321986+0000","last_undegraded":"2021-01-15T18:46:48.321986+0000","last_fullsized":"2021-01-15T18:46:48.321986+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6645,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23318635,"num_objects":2848,"num_object_clones":0,"num_object_copies":11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6644,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6822","reported_seq":"7076","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.072411+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:50.072411+0000","last_peered":"2021-01-15T18:46:50.072411+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:50.072411+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6822,"ondisk_log_size":6822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24330240,"num_objects":2970,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6822,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6910","reported_seq":"7510","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.112218+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:50.112218+0000","last_peered":"2021-01-15T18:46:50.112218+0000","last_clean":"2021-01-15T18:46:50.112218+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:50.112218+0000","last_undegraded":"2021-01-15T18:46:50.112218+0000","last_fullsized":"2021-01-15T18:46:50.112218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6910,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23904422,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6592","reported_seq":"6811","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.056000+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:50.056000+0000","last_peered":"2021-01-15T18:46:50.056000+0000","last_clean":"2021-01-15T18:46:50.056000+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:50.056000+0000","last_undegraded":"2021-01-15T18:46:50.056000+0000","last_fullsized":"2021-01-15T18:46:50.056000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24035494,"num_objects":2935,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6590,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6758","reported_seq":"7512","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.115096+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:50.115096+0000","last_peered":"2021-01-15T18:46:50.115096+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:50.115096+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6758,"ondisk_log_size":6758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24305830,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6881","reported_seq":"7045","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:48.614833+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:48.614833+0000","last_peered":"2021-01-15T18:46:48.614833+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:48.614833+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6881,"ondisk_log_size":6881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24375379,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1563}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6798","reported_seq":"9741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.055515+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:50.055515+0000","last_peered":"2021-01-15T18:46:50.055515+0000","last_clean":"2021-01-15T18:46:50.055515+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:50.055515+0000","last_undegraded":"2021-01-15T18:46:50.055515+0000","last_fullsized":"2021-01-15T18:46:50.055515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6798,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24006739,"num_objects":2931,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6829","reported_seq":"9777","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.092709+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:50.092709+0000","last_peered":"2021-01-15T18:46:50.092709+0000","last_clean":"2021-01-15T18:46:50.092709+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:50.092709+0000","last_undegraded":"2021-01-15T18:46:50.092709+0000","last_fullsized":"2021-01-15T18:46:50.092709+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23990521,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6610","reported_seq":"6858","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:49.929975+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:49.929975+0000","last_peered":"2021-01-15T18:46:49.929975+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:49.929975+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6610,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24117414,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6644","reported_seq":"9481","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.947902+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:49.947902+0000","last_peered":"2021-01-15T18:46:49.947902+0000","last_clean":"2021-01-15T18:46:49.947902+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:49.947902+0000","last_undegraded":"2021-01-15T18:46:49.947902+0000","last_fullsized":"2021-01-15T18:46:49.947902+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6644,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23945382,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6607","reported_seq":"9510","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.281261+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:49.281261+0000","last_peered":"2021-01-15T18:46:49.281261+0000","last_clean":"2021-01-15T18:46:49.281261+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:49.281261+0000","last_undegraded":"2021-01-15T18:46:49.281261+0000","last_fullsized":"2021-01-15T18:46:49.281261+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6607,"ondisk_log_size":6607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23494822,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6605,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'6926","reported_seq":"7160","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.772566+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:49.772566+0000","last_peered":"2021-01-15T18:46:49.772566+0000","last_clean":"2021-01-15T18:46:49.772566+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:49.772566+0000","last_undegraded":"2021-01-15T18:46:49.772566+0000","last_fullsized":"2021-01-15T18:46:49.772566+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6926,"ondisk_log_size":6926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248486,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6924,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7032","reported_seq":"7304","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.025573+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:50.025573+0000","last_peered":"2021-01-15T18:46:50.025573+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:50.025573+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7032,"ondisk_log_size":7032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24658252,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6626","reported_seq":"9499","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:48.512288+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:48.512288+0000","last_peered":"2021-01-15T18:46:48.512288+0000","last_clean":"2021-01-15T18:46:48.512288+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:48.512288+0000","last_undegraded":"2021-01-15T18:46:48.512288+0000","last_fullsized":"2021-01-15T18:46:48.512288+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6626,"ondisk_log_size":6626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23863462,"num_objects":2914,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":383887660,"num_objects":46876,"num_object_clones":0,"num_object_copies":187504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107849,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":768462848,"data_stored":768462848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107876,"ondisk_log_size":107876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400204,"kb_used_data":188040,"kb_used_omap":0,"kb_used_meta":212095,"kb_avail":93971636,"statfs":{"total":96636764160,"available":96226955264,"internally_reserved":0,"allocated":192552960,"data_stored":192299782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":217185332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":24.763000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":12.958}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":5.9109999999999996},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":6.6399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401100,"kb_used_data":189472,"kb_used_omap":0,"kb_used_meta":211584,"kb_avail":93970740,"statfs":{"total":96636764160,"available":96226037760,"internally_reserved":0,"allocated":194019328,"data_stored":193766150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":216662016},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,0,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":5.6189999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":10.971},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":5.9050000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":5.7309999999999999},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":0.77000000000000002}]}]},{"osd":0,"up_from":22,"seq":94489280604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392200,"kb_used_data":188304,"kb_used_omap":0,"kb_used_meta":203839,"kb_avail":93979640,"statfs":{"total":96636764160,"available":96235151360,"internally_reserved":0,"allocated":192823296,"data_stored":192570118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":208731188},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":2.6920000000000002},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":17.661000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":0.79200000000000004},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":8.3109999999999999},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.51300000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388968,"kb_used_data":188312,"kb_used_omap":0,"kb_used_meta":200575,"kb_avail":93982872,"statfs":{"total":96636764160,"available":96238460928,"internally_reserved":0,"allocated":192831488,"data_stored":192578310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":205388852},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.503},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.90000000000000002}]}]}]}} 2021-01-15T18:46:53.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:54.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:54.816 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:54.850 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2021-01-15T18:46:54.579391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381462828,"num_objects":46580,"num_object_clones":0,"num_object_copies":186320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108145,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108172,"ondisk_log_size":108172,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593980,"kb_used_data":745100,"kb_used_omap":2,"kb_used_meta":848637,"kb_avail":375893380,"statfs":{"total":386547056640,"available":384914821120,"internally_reserved":0,"allocated":762982400,"data_stored":761969688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":869004444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,13,16,15,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20307968,"num_objects":-2479,"num_object_clones":0,"num_object_copies":-9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009848"},"pg_stats":[{"pgid":"1.f","version":"48'6581","reported_seq":"6847","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.072243+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:50.072243+0000","last_peered":"2021-01-15T18:46:50.072243+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:50.072243+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6581,"ondisk_log_size":6581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23977984,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6581,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6615","reported_seq":"9547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.989178+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:49.989178+0000","last_peered":"2021-01-15T18:46:49.989178+0000","last_clean":"2021-01-15T18:46:49.989178+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:49.989178+0000","last_undegraded":"2021-01-15T18:46:49.989178+0000","last_fullsized":"2021-01-15T18:46:49.989178+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6615,"ondisk_log_size":6615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23314598,"num_objects":2847,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6613,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6710","reported_seq":"9649","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:53.568045+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:53.568045+0000","last_peered":"2021-01-15T18:46:53.568045+0000","last_clean":"2021-01-15T18:46:53.568045+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:53.568045+0000","last_undegraded":"2021-01-15T18:46:53.568045+0000","last_fullsized":"2021-01-15T18:46:53.568045+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22786155,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6822","reported_seq":"7076","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.072411+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:50.072411+0000","last_peered":"2021-01-15T18:46:50.072411+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:50.072411+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6822,"ondisk_log_size":6822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24330240,"num_objects":2970,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6822,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6910","reported_seq":"7510","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.112218+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:50.112218+0000","last_peered":"2021-01-15T18:46:50.112218+0000","last_clean":"2021-01-15T18:46:50.112218+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:50.112218+0000","last_undegraded":"2021-01-15T18:46:50.112218+0000","last_fullsized":"2021-01-15T18:46:50.112218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6910,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23904422,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6592","reported_seq":"6811","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.056000+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:50.056000+0000","last_peered":"2021-01-15T18:46:50.056000+0000","last_clean":"2021-01-15T18:46:50.056000+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:50.056000+0000","last_undegraded":"2021-01-15T18:46:50.056000+0000","last_fullsized":"2021-01-15T18:46:50.056000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24035494,"num_objects":2935,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6590,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6758","reported_seq":"7512","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.115096+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:50.115096+0000","last_peered":"2021-01-15T18:46:50.115096+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:50.115096+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6758,"ondisk_log_size":6758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24305830,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1918}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6960","reported_seq":"7123","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:53.587940+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:53.587940+0000","last_peered":"2021-01-15T18:46:53.587940+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:53.587940+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6960,"ondisk_log_size":6960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23728211,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6798","reported_seq":"9741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.055515+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:50.055515+0000","last_peered":"2021-01-15T18:46:50.055515+0000","last_clean":"2021-01-15T18:46:50.055515+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:50.055515+0000","last_undegraded":"2021-01-15T18:46:50.055515+0000","last_fullsized":"2021-01-15T18:46:50.055515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6798,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24006739,"num_objects":2931,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6829","reported_seq":"9777","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:50.092709+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:50.092709+0000","last_peered":"2021-01-15T18:46:50.092709+0000","last_clean":"2021-01-15T18:46:50.092709+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:50.092709+0000","last_undegraded":"2021-01-15T18:46:50.092709+0000","last_fullsized":"2021-01-15T18:46:50.092709+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6829,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23990521,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6610","reported_seq":"6858","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:49.929975+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:49.929975+0000","last_peered":"2021-01-15T18:46:49.929975+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:49.929975+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6610,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24117414,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6644","reported_seq":"9481","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.947902+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:49.947902+0000","last_peered":"2021-01-15T18:46:49.947902+0000","last_clean":"2021-01-15T18:46:49.947902+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:49.947902+0000","last_undegraded":"2021-01-15T18:46:49.947902+0000","last_fullsized":"2021-01-15T18:46:49.947902+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6644,"ondisk_log_size":6644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23945382,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6677","reported_seq":"9580","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:54.405558+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:54.405558+0000","last_peered":"2021-01-15T18:46:54.405558+0000","last_clean":"2021-01-15T18:46:54.405558+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:54.405558+0000","last_undegraded":"2021-01-15T18:46:54.405558+0000","last_fullsized":"2021-01-15T18:46:54.405558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6677,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22921382,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'6926","reported_seq":"7160","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:49.772566+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:49.772566+0000","last_peered":"2021-01-15T18:46:49.772566+0000","last_clean":"2021-01-15T18:46:49.772566+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:49.772566+0000","last_undegraded":"2021-01-15T18:46:49.772566+0000","last_fullsized":"2021-01-15T18:46:49.772566+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6926,"ondisk_log_size":6926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248486,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6924,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7032","reported_seq":"7304","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:50.025573+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:50.025573+0000","last_peered":"2021-01-15T18:46:50.025573+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:50.025573+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7032,"ondisk_log_size":7032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24658252,"num_objects":3012,"num_object_clones":0,"num_object_copies":12048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6708","reported_seq":"9581","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:53.449866+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:53.449866+0000","last_peered":"2021-01-15T18:46:53.449866+0000","last_clean":"2021-01-15T18:46:53.449866+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:53.449866+0000","last_undegraded":"2021-01-15T18:46:53.449866+0000","last_fullsized":"2021-01-15T18:46:53.449866+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6708,"ondisk_log_size":6708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23191718,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":381462828,"num_objects":46580,"num_object_clones":0,"num_object_copies":186320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7730,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108145,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13677,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":759332864,"data_stored":759332864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108172,"ondisk_log_size":108172,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215192,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400204,"kb_used_data":188040,"kb_used_omap":0,"kb_used_meta":212095,"kb_avail":93971636,"statfs":{"total":96636764160,"available":96226955264,"internally_reserved":0,"allocated":192552960,"data_stored":192299782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":217185332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":24.763000000000002},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":12.958}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":5.9109999999999996},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":6.6399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406808,"kb_used_data":184900,"kb_used_omap":0,"kb_used_meta":221888,"kb_avail":93965032,"statfs":{"total":96636764160,"available":96220192768,"internally_reserved":0,"allocated":189337600,"data_stored":189084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":227213312},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,5,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.78500000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.125}]}]},{"osd":0,"up_from":22,"seq":94489280605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398000,"kb_used_data":183848,"kb_used_omap":0,"kb_used_meta":214079,"kb_avail":93973840,"statfs":{"total":96636764160,"available":96229212160,"internally_reserved":0,"allocated":188260352,"data_stored":188007174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":219216948},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":2.0419999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.7839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.59499999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388968,"kb_used_data":188312,"kb_used_omap":0,"kb_used_meta":200575,"kb_avail":93982872,"statfs":{"total":96636764160,"available":96238460928,"internally_reserved":0,"allocated":192831488,"data_stored":192578310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":205388852},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.503},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.90000000000000002}]}]}]}} 2021-01-15T18:46:54.852 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:46:55.498 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:46:55.495+0000 7fb198fcd700 0 --2- 172.21.15.148:0/2237663645 >> v2:172.21.15.148:3300/0 conn(0x7fb1941282b0 0x7fb19412a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-01-15T18:46:56.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:56.282 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7730/186320 objects degraded (4.149%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46580,"data_bytes":381462828,"bytes_used":1632235520,"bytes_avail":384914821120,"bytes_total":386547056640,"degraded_objects":7730,"degraded_total":186320,"degraded_ratio":0.041487762988407041,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:46:56.359 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:46:56.360 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:46:57.661 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:46:57.703 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:46:57.757 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2021-01-15T18:46:56.579809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374712620,"num_objects":45756,"num_object_clones":0,"num_object_copies":183024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108969,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13729,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108996,"ondisk_log_size":108996,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1605476,"kb_used_data":736096,"kb_used_omap":2,"kb_used_meta":869245,"kb_avail":375881884,"statfs":{"total":386547056640,"available":384903049216,"internally_reserved":0,"allocated":753762304,"data_stored":752749592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":890107036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,10,15,18,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":119,"apply_latency_ms":119,"commit_latency_ns":119000000,"apply_latency_ns":119000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24854528,"num_objects":-3034,"num_object_clones":0,"num_object_copies":-12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009851"},"pg_stats":[{"pgid":"1.f","version":"48'6643","reported_seq":"6909","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:54.967642+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:46:54.967642+0000","last_peered":"2021-01-15T18:46:54.967642+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:46:54.967642+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6643,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23470080,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6682","reported_seq":"9614","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:55.234072+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:55.234072+0000","last_peered":"2021-01-15T18:46:55.234072+0000","last_clean":"2021-01-15T18:46:55.234072+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:55.234072+0000","last_undegraded":"2021-01-15T18:46:55.234072+0000","last_fullsized":"2021-01-15T18:46:55.234072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6682,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22765734,"num_objects":2780,"num_object_clones":0,"num_object_copies":11120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6710","reported_seq":"9649","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:53.568045+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:53.568045+0000","last_peered":"2021-01-15T18:46:53.568045+0000","last_clean":"2021-01-15T18:46:53.568045+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:53.568045+0000","last_undegraded":"2021-01-15T18:46:53.568045+0000","last_fullsized":"2021-01-15T18:46:53.568045+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22786155,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6709,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6895","reported_seq":"7149","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:54.901444+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:46:54.901444+0000","last_peered":"2021-01-15T18:46:54.901444+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:46:54.901444+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6895,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23732224,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'6964","reported_seq":"7564","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:54.996290+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:46:54.996290+0000","last_peered":"2021-01-15T18:46:54.996290+0000","last_clean":"2021-01-15T18:46:54.996290+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:46:54.996290+0000","last_undegraded":"2021-01-15T18:46:54.996290+0000","last_fullsized":"2021-01-15T18:46:54.996290+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6964,"ondisk_log_size":6964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23462054,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6667","reported_seq":"6886","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:55.074639+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:55.074639+0000","last_peered":"2021-01-15T18:46:55.074639+0000","last_clean":"2021-01-15T18:46:55.074639+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:55.074639+0000","last_undegraded":"2021-01-15T18:46:55.074639+0000","last_fullsized":"2021-01-15T18:46:55.074639+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6667,"ondisk_log_size":6667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23421094,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6823","reported_seq":"7680","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:55.145439+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:46:55.145439+0000","last_peered":"2021-01-15T18:46:55.145439+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:46:55.145439+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6823,"ondisk_log_size":6823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23773350,"num_objects":2903,"num_object_clones":0,"num_object_copies":11612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1905}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'6960","reported_seq":"7123","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:53.587940+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:53.587940+0000","last_peered":"2021-01-15T18:46:53.587940+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:53.587940+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6960,"ondisk_log_size":6960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23728211,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6870","reported_seq":"9812","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:55.172905+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:46:55.172905+0000","last_peered":"2021-01-15T18:46:55.172905+0000","last_clean":"2021-01-15T18:46:55.172905+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:46:55.172905+0000","last_undegraded":"2021-01-15T18:46:55.172905+0000","last_fullsized":"2021-01-15T18:46:55.172905+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6870,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23416915,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6869,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6896","reported_seq":"9845","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:54.921230+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:46:54.921230+0000","last_peered":"2021-01-15T18:46:54.921230+0000","last_clean":"2021-01-15T18:46:54.921230+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:46:54.921230+0000","last_undegraded":"2021-01-15T18:46:54.921230+0000","last_fullsized":"2021-01-15T18:46:54.921230+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6896,"ondisk_log_size":6896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23441657,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6893,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6687","reported_seq":"6935","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:55.088187+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:46:55.088187+0000","last_peered":"2021-01-15T18:46:55.088187+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:46:55.088187+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23486630,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1576}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6701","reported_seq":"9538","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:55.169120+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:46:55.169120+0000","last_peered":"2021-01-15T18:46:55.169120+0000","last_clean":"2021-01-15T18:46:55.169120+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:46:55.169120+0000","last_undegraded":"2021-01-15T18:46:55.169120+0000","last_fullsized":"2021-01-15T18:46:55.169120+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6701,"ondisk_log_size":6701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23478438,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6699,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6677","reported_seq":"9580","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:54.405558+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:54.405558+0000","last_peered":"2021-01-15T18:46:54.405558+0000","last_clean":"2021-01-15T18:46:54.405558+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:54.405558+0000","last_undegraded":"2021-01-15T18:46:54.405558+0000","last_fullsized":"2021-01-15T18:46:54.405558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6677,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22921382,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7001","reported_seq":"7235","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:55.191064+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:46:55.191064+0000","last_peered":"2021-01-15T18:46:55.191064+0000","last_clean":"2021-01-15T18:46:55.191064+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:46:55.191064+0000","last_undegraded":"2021-01-15T18:46:55.191064+0000","last_fullsized":"2021-01-15T18:46:55.191064+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7001,"ondisk_log_size":7001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23634086,"num_objects":2886,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7112","reported_seq":"7384","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:55.089959+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:46:55.089959+0000","last_peered":"2021-01-15T18:46:55.089959+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:46:55.089959+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7112,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24002892,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6708","reported_seq":"9581","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:53.449866+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:53.449866+0000","last_peered":"2021-01-15T18:46:53.449866+0000","last_clean":"2021-01-15T18:46:53.449866+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:53.449866+0000","last_undegraded":"2021-01-15T18:46:53.449866+0000","last_fullsized":"2021-01-15T18:46:53.449866+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6708,"ondisk_log_size":6708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23191718,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374712620,"num_objects":45756,"num_object_clones":0,"num_object_copies":183024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108969,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13729,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":750194688,"data_stored":750194688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108996,"ondisk_log_size":108996,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215193,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405928,"kb_used_data":183560,"kb_used_omap":0,"kb_used_meta":222335,"kb_avail":93965912,"statfs":{"total":96636764160,"available":96221093888,"internally_reserved":0,"allocated":187965440,"data_stored":187712262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":227671092},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.2350000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":8.827},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":8.9250000000000007}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":1},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":11.228}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406808,"kb_used_data":184900,"kb_used_omap":0,"kb_used_meta":221888,"kb_avail":93965032,"statfs":{"total":96636764160,"available":96220192768,"internally_reserved":0,"allocated":189337600,"data_stored":189084422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":227213312},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,5,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":3.0470000000000002,"5min":3.6779999999999999,"15min":3.7290000000000001},"min":{"1min":0.40600000000000003,"5min":0.35899999999999999,"15min":0.33800000000000002},"max":{"1min":27.870000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.976,"5min":2.9500000000000002,"15min":1.6399999999999999},"min":{"1min":0.41099999999999998,"5min":0.371,"15min":0.371},"max":{"1min":15.721,"5min":23.98,"15min":23.98},"last":0.78500000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:45:56 2021","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":3.0630000000000002,"15min":2.5470000000000002},"min":{"1min":0.53100000000000003,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.904,"5min":21.952000000000002,"15min":21.952000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":2.48,"5min":3.1819999999999999,"15min":2.5750000000000002},"min":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":15.856999999999999,"5min":18.34,"15min":18.34},"last":1.125}]}]},{"osd":0,"up_from":22,"seq":94489280605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398000,"kb_used_data":183848,"kb_used_omap":0,"kb_used_meta":214079,"kb_avail":93973840,"statfs":{"total":96636764160,"available":96229212160,"internally_reserved":0,"allocated":188260352,"data_stored":188007174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":219216948},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:45:51 2021","interfaces":[{"interface":"back","average":{"1min":2.6539999999999999,"5min":3.3109999999999999,"15min":2.5830000000000002},"min":{"1min":0.434,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":9.7759999999999998,"5min":27.77,"15min":27.77},"last":2.0419999999999998},{"interface":"front","average":{"1min":2.367,"5min":3.2120000000000002,"15min":2.6179999999999999},"min":{"1min":0.56899999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.9979999999999993,"5min":22.765000000000001,"15min":22.765000000000001},"last":4.7839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":0.59499999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394740,"kb_used_data":183788,"kb_used_omap":0,"kb_used_meta":210943,"kb_avail":93977100,"statfs":{"total":96636764160,"available":96232550400,"internally_reserved":0,"allocated":188198912,"data_stored":187945734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":216005684},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,7,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.645},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.573}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.4470000000000001},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":2.548}]}]}]}} 2021-01-15T18:47:00.759 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:01.770 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:01.773 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:01.785 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2021-01-15T18:47:00.582509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365455660,"num_objects":44626,"num_object_clones":0,"num_object_copies":178504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110099,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13790,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110126,"ondisk_log_size":110126,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1554868,"kb_used_data":718216,"kb_used_omap":2,"kb_used_meta":836541,"kb_avail":375932492,"statfs":{"total":386547056640,"available":384954871808,"internally_reserved":0,"allocated":735453184,"data_stored":734440472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":856618140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,5,5,4,20,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27541504,"num_objects":-3362,"num_object_clones":0,"num_object_copies":-13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007451"},"pg_stats":[{"pgid":"1.f","version":"48'6700","reported_seq":"6966","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.046723+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:00.046723+0000","last_peered":"2021-01-15T18:47:00.046723+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:00.046723+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6700,"ondisk_log_size":6700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23003136,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6700,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6756","reported_seq":"9688","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.979365+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:59.979365+0000","last_peered":"2021-01-15T18:46:59.979365+0000","last_clean":"2021-01-15T18:46:59.979365+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:59.979365+0000","last_undegraded":"2021-01-15T18:46:59.979365+0000","last_fullsized":"2021-01-15T18:46:59.979365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6756,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22159526,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6772","reported_seq":"9711","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:58.614583+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:58.614583+0000","last_peered":"2021-01-15T18:46:58.614583+0000","last_clean":"2021-01-15T18:46:58.614583+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:58.614583+0000","last_undegraded":"2021-01-15T18:46:58.614583+0000","last_fullsized":"2021-01-15T18:46:58.614583+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22278251,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6771,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6967","reported_seq":"7220","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.105590+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:00.105590+0000","last_peered":"2021-01-15T18:47:00.105590+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:00.105590+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6967,"ondisk_log_size":6967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23142400,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7037","reported_seq":"7637","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.096708+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:00.096708+0000","last_peered":"2021-01-15T18:47:00.096708+0000","last_clean":"2021-01-15T18:47:00.096708+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:00.096708+0000","last_undegraded":"2021-01-15T18:47:00.096708+0000","last_fullsized":"2021-01-15T18:47:00.096708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7037,"ondisk_log_size":7037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22864038,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7035,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6748","reported_seq":"6966","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.829259+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:59.829259+0000","last_peered":"2021-01-15T18:46:59.829259+0000","last_clean":"2021-01-15T18:46:59.829259+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:59.829259+0000","last_undegraded":"2021-01-15T18:46:59.829259+0000","last_fullsized":"2021-01-15T18:46:59.829259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6748,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22757542,"num_objects":2779,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6746,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6885","reported_seq":"7866","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.156531+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:00.156531+0000","last_peered":"2021-01-15T18:47:00.156531+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:00.156531+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6885,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23265446,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1904}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7024","reported_seq":"7188","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:58.632514+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:58.632514+0000","last_peered":"2021-01-15T18:46:58.632514+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:58.632514+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7024,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23203923,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6941","reported_seq":"9882","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.235146+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:00.235146+0000","last_peered":"2021-01-15T18:47:00.235146+0000","last_clean":"2021-01-15T18:47:00.235146+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:00.235146+0000","last_undegraded":"2021-01-15T18:47:00.235146+0000","last_fullsized":"2021-01-15T18:47:00.235146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6941,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22835283,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6940,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6967","reported_seq":"9916","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.038797+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:00.038797+0000","last_peered":"2021-01-15T18:47:00.038797+0000","last_clean":"2021-01-15T18:47:00.038797+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:00.038797+0000","last_undegraded":"2021-01-15T18:47:00.038797+0000","last_fullsized":"2021-01-15T18:47:00.038797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6967,"ondisk_log_size":6967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22860025,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6964,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6756","reported_seq":"7003","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.152723+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:00.152723+0000","last_peered":"2021-01-15T18:47:00.152723+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:00.152723+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6756,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22921382,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6777","reported_seq":"9614","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.218269+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:00.218269+0000","last_peered":"2021-01-15T18:47:00.218269+0000","last_clean":"2021-01-15T18:47:00.218269+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:00.218269+0000","last_undegraded":"2021-01-15T18:47:00.218269+0000","last_fullsized":"2021-01-15T18:47:00.218269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6777,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22855846,"num_objects":2791,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6747","reported_seq":"9649","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.313749+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:59.313749+0000","last_peered":"2021-01-15T18:46:59.313749+0000","last_clean":"2021-01-15T18:46:59.313749+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:59.313749+0000","last_undegraded":"2021-01-15T18:46:59.313749+0000","last_fullsized":"2021-01-15T18:46:59.313749+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6747,"ondisk_log_size":6747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22347942,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7079","reported_seq":"7313","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.134142+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:00.134142+0000","last_peered":"2021-01-15T18:47:00.134142+0000","last_clean":"2021-01-15T18:47:00.134142+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:00.134142+0000","last_undegraded":"2021-01-15T18:47:00.134142+0000","last_fullsized":"2021-01-15T18:47:00.134142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7079,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22995110,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7186","reported_seq":"7460","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.194423+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:00.194423+0000","last_peered":"2021-01-15T18:47:00.194423+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:00.194423+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7186,"ondisk_log_size":7186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23396684,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6784","reported_seq":"9657","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:58.471520+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:58.471520+0000","last_peered":"2021-01-15T18:46:58.471520+0000","last_clean":"2021-01-15T18:46:58.471520+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:58.471520+0000","last_undegraded":"2021-01-15T18:46:58.471520+0000","last_fullsized":"2021-01-15T18:46:58.471520+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6784,"ondisk_log_size":6784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22569126,"num_objects":2756,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365455660,"num_objects":44626,"num_object_clones":0,"num_object_copies":178504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110099,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13790,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":731844608,"data_stored":731844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110126,"ondisk_log_size":110126,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215194,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391232,"kb_used_data":179112,"kb_used_omap":0,"kb_used_meta":212095,"kb_avail":93980608,"statfs":{"total":96636764160,"available":96236142592,"internally_reserved":0,"allocated":183410688,"data_stored":183157510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":217185332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,2,0,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":2.109},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.9630000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":14.776999999999999},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":2.0169999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396220,"kb_used_data":180532,"kb_used_omap":0,"kb_used_meta":215680,"kb_avail":93975620,"statfs":{"total":96636764160,"available":96231034880,"internally_reserved":0,"allocated":184864768,"data_stored":184611590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220856320},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.0569999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.3420000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.92200000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.878},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.8280000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387328,"kb_used_data":179340,"kb_used_omap":0,"kb_used_meta":207935,"kb_avail":93984512,"statfs":{"total":96636764160,"available":96240140288,"internally_reserved":0,"allocated":183644160,"data_stored":183390982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":212925492},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":20.783000000000001},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":25.754999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":8.7910000000000004},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":11.595000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":7.3440000000000003},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":7.8659999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380088,"kb_used_data":179232,"kb_used_omap":0,"kb_used_meta":200831,"kb_avail":93991752,"statfs":{"total":96636764160,"available":96247554048,"internally_reserved":0,"allocated":183533568,"data_stored":183280390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":205650996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.042}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.749},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.93100000000000005}]}]}]}} 2021-01-15T18:47:01.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:02.822 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:02.822 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7617/178504 objects degraded (4.267%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":473,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44626,"data_bytes":365455660,"bytes_used":1592184832,"bytes_avail":384954871808,"bytes_total":386547056640,"degraded_objects":7617,"degraded_total":178504,"degraded_ratio":0.042671312687670868,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":279},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:02.844 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:02.845 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:03.480 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:03.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2021-01-15T18:47:02.582989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":365455660,"num_objects":44626,"num_object_clones":0,"num_object_copies":178504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110099,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13790,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110126,"ondisk_log_size":110126,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1554868,"kb_used_data":718216,"kb_used_omap":2,"kb_used_meta":836541,"kb_avail":375932492,"statfs":{"total":386547056640,"available":384954871808,"internally_reserved":0,"allocated":735453184,"data_stored":734440472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":856618140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,5,5,4,20,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18432000,"num_objects":-2250,"num_object_clones":0,"num_object_copies":-9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007328"},"pg_stats":[{"pgid":"1.f","version":"48'6700","reported_seq":"6966","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.046723+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:00.046723+0000","last_peered":"2021-01-15T18:47:00.046723+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:00.046723+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6700,"ondisk_log_size":6700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23003136,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6700,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6756","reported_seq":"9688","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.979365+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:46:59.979365+0000","last_peered":"2021-01-15T18:46:59.979365+0000","last_clean":"2021-01-15T18:46:59.979365+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:46:59.979365+0000","last_undegraded":"2021-01-15T18:46:59.979365+0000","last_fullsized":"2021-01-15T18:46:59.979365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6756,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22159526,"num_objects":2706,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6772","reported_seq":"9711","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:58.614583+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:46:58.614583+0000","last_peered":"2021-01-15T18:46:58.614583+0000","last_clean":"2021-01-15T18:46:58.614583+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:46:58.614583+0000","last_undegraded":"2021-01-15T18:46:58.614583+0000","last_fullsized":"2021-01-15T18:46:58.614583+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22278251,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6771,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'6967","reported_seq":"7220","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.105590+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:00.105590+0000","last_peered":"2021-01-15T18:47:00.105590+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:00.105590+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6967,"ondisk_log_size":6967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23142400,"num_objects":2825,"num_object_clones":0,"num_object_copies":11300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7037","reported_seq":"7637","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.096708+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:00.096708+0000","last_peered":"2021-01-15T18:47:00.096708+0000","last_clean":"2021-01-15T18:47:00.096708+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:00.096708+0000","last_undegraded":"2021-01-15T18:47:00.096708+0000","last_fullsized":"2021-01-15T18:47:00.096708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7037,"ondisk_log_size":7037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22864038,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7035,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6748","reported_seq":"6966","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.829259+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:46:59.829259+0000","last_peered":"2021-01-15T18:46:59.829259+0000","last_clean":"2021-01-15T18:46:59.829259+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:46:59.829259+0000","last_undegraded":"2021-01-15T18:46:59.829259+0000","last_fullsized":"2021-01-15T18:46:59.829259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6748,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22757542,"num_objects":2779,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6746,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6885","reported_seq":"7866","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.156531+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:00.156531+0000","last_peered":"2021-01-15T18:47:00.156531+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:00.156531+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6885,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23265446,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1904}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7024","reported_seq":"7188","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:46:58.632514+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:46:58.632514+0000","last_peered":"2021-01-15T18:46:58.632514+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:46:58.632514+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7024,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23203923,"num_objects":2833,"num_object_clones":0,"num_object_copies":11332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'6941","reported_seq":"9882","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.235146+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:00.235146+0000","last_peered":"2021-01-15T18:47:00.235146+0000","last_clean":"2021-01-15T18:47:00.235146+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:00.235146+0000","last_undegraded":"2021-01-15T18:47:00.235146+0000","last_fullsized":"2021-01-15T18:47:00.235146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6941,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22835283,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6940,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'6967","reported_seq":"9916","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.038797+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:00.038797+0000","last_peered":"2021-01-15T18:47:00.038797+0000","last_clean":"2021-01-15T18:47:00.038797+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:00.038797+0000","last_undegraded":"2021-01-15T18:47:00.038797+0000","last_fullsized":"2021-01-15T18:47:00.038797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6967,"ondisk_log_size":6967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22860025,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6964,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6756","reported_seq":"7003","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.152723+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:00.152723+0000","last_peered":"2021-01-15T18:47:00.152723+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:00.152723+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6756,"ondisk_log_size":6756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22921382,"num_objects":2799,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6777","reported_seq":"9614","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.218269+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:00.218269+0000","last_peered":"2021-01-15T18:47:00.218269+0000","last_clean":"2021-01-15T18:47:00.218269+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:00.218269+0000","last_undegraded":"2021-01-15T18:47:00.218269+0000","last_fullsized":"2021-01-15T18:47:00.218269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6777,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22855846,"num_objects":2791,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6747","reported_seq":"9649","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:59.313749+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:46:59.313749+0000","last_peered":"2021-01-15T18:46:59.313749+0000","last_clean":"2021-01-15T18:46:59.313749+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:46:59.313749+0000","last_undegraded":"2021-01-15T18:46:59.313749+0000","last_fullsized":"2021-01-15T18:46:59.313749+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6747,"ondisk_log_size":6747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22347942,"num_objects":2729,"num_object_clones":0,"num_object_copies":10916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7079","reported_seq":"7313","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:00.134142+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:00.134142+0000","last_peered":"2021-01-15T18:47:00.134142+0000","last_clean":"2021-01-15T18:47:00.134142+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:00.134142+0000","last_undegraded":"2021-01-15T18:47:00.134142+0000","last_fullsized":"2021-01-15T18:47:00.134142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7079,"ondisk_log_size":7079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22995110,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7186","reported_seq":"7460","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:00.194423+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:00.194423+0000","last_peered":"2021-01-15T18:47:00.194423+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:00.194423+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7186,"ondisk_log_size":7186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23396684,"num_objects":2858,"num_object_clones":0,"num_object_copies":11432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6784","reported_seq":"9657","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:46:58.471520+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:46:58.471520+0000","last_peered":"2021-01-15T18:46:58.471520+0000","last_clean":"2021-01-15T18:46:58.471520+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:46:58.471520+0000","last_undegraded":"2021-01-15T18:46:58.471520+0000","last_fullsized":"2021-01-15T18:46:58.471520+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6784,"ondisk_log_size":6784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22569126,"num_objects":2756,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":365455660,"num_objects":44626,"num_object_clones":0,"num_object_copies":178504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110099,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13790,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":731844608,"data_stored":731844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110126,"ondisk_log_size":110126,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215194,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391232,"kb_used_data":179112,"kb_used_omap":0,"kb_used_meta":212095,"kb_avail":93980608,"statfs":{"total":96636764160,"available":96236142592,"internally_reserved":0,"allocated":183410688,"data_stored":183157510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":217185332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,2,0,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":2.109},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.9630000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":14.776999999999999},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":2.0169999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396220,"kb_used_data":180532,"kb_used_omap":0,"kb_used_meta":215680,"kb_avail":93975620,"statfs":{"total":96636764160,"available":96231034880,"internally_reserved":0,"allocated":184864768,"data_stored":184611590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220856320},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.0569999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.3420000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.92200000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.878},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.8280000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387328,"kb_used_data":179340,"kb_used_omap":0,"kb_used_meta":207935,"kb_avail":93984512,"statfs":{"total":96636764160,"available":96240140288,"internally_reserved":0,"allocated":183644160,"data_stored":183390982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":212925492},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":20.783000000000001},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":25.754999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":8.7910000000000004},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":11.595000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:45:59 2021","interfaces":[{"interface":"back","average":{"1min":4.8129999999999997,"5min":3.5590000000000002,"15min":4.3150000000000004},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":36.204999999999998,"5min":36.204999999999998,"15min":41.984999999999999},"last":7.3440000000000003},{"interface":"front","average":{"1min":2.8460000000000001,"5min":4.8230000000000004,"15min":3.028},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48199999999999998},"max":{"1min":10.326000000000001,"5min":56.927,"15min":56.927},"last":7.8659999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380088,"kb_used_data":179232,"kb_used_omap":0,"kb_used_meta":200831,"kb_avail":93991752,"statfs":{"total":96636764160,"available":96247554048,"internally_reserved":0,"allocated":183533568,"data_stored":183280390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":205650996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.042}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.749},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.93100000000000005}]}]}]}} 2021-01-15T18:47:06.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:07.737 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:07.749 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:07.828 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2021-01-15T18:47:06.583805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357034284,"num_objects":43598,"num_object_clones":0,"num_object_copies":174392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111127,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13854,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111154,"ondisk_log_size":111154,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1574760,"kb_used_data":701684,"kb_used_omap":2,"kb_used_meta":872893,"kb_avail":375912600,"statfs":{"total":386547056640,"available":384934502400,"internally_reserved":0,"allocated":718524416,"data_stored":717511704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":893842588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,4,7,13,22,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24428544,"num_objects":-2982,"num_object_clones":0,"num_object_copies":-11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004414"},"pg_stats":[{"pgid":"1.f","version":"48'6759","reported_seq":"7025","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.083414+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:05.083414+0000","last_peered":"2021-01-15T18:47:05.083414+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:05.083414+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6759,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22519808,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6759,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6809","reported_seq":"9741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.207087+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:05.207087+0000","last_peered":"2021-01-15T18:47:05.207087+0000","last_clean":"2021-01-15T18:47:05.207087+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:05.207087+0000","last_undegraded":"2021-01-15T18:47:05.207087+0000","last_fullsized":"2021-01-15T18:47:05.207087+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21725350,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6833","reported_seq":"9772","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:03.358234+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:03.358234+0000","last_peered":"2021-01-15T18:47:03.358234+0000","last_clean":"2021-01-15T18:47:03.358234+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:03.358234+0000","last_undegraded":"2021-01-15T18:47:03.358234+0000","last_fullsized":"2021-01-15T18:47:03.358234+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6833,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21778539,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6832,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7045","reported_seq":"7299","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:04.847105+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:04.847105+0000","last_peered":"2021-01-15T18:47:04.847105+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:04.847105+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7045,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22503424,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7097","reported_seq":"7697","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.274715+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:05.274715+0000","last_peered":"2021-01-15T18:47:05.274715+0000","last_clean":"2021-01-15T18:47:05.274715+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:05.274715+0000","last_undegraded":"2021-01-15T18:47:05.274715+0000","last_fullsized":"2021-01-15T18:47:05.274715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7097,"ondisk_log_size":7097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22372518,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6820","reported_seq":"7039","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.185751+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:05.185751+0000","last_peered":"2021-01-15T18:47:05.185751+0000","last_clean":"2021-01-15T18:47:05.185751+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:05.185751+0000","last_undegraded":"2021-01-15T18:47:05.185751+0000","last_fullsized":"2021-01-15T18:47:05.185751+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6820,"ondisk_log_size":6820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22167718,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6818,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6965","reported_seq":"8074","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.246283+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:05.246283+0000","last_peered":"2021-01-15T18:47:05.246283+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:05.246283+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6965,"ondisk_log_size":6965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22610086,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7078","reported_seq":"7242","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:03.419982+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:03.419982+0000","last_peered":"2021-01-15T18:47:03.419982+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:03.419982+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7078,"ondisk_log_size":7078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22761555,"num_objects":2779,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7010","reported_seq":"9953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.124028+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:05.124028+0000","last_peered":"2021-01-15T18:47:05.124028+0000","last_clean":"2021-01-15T18:47:05.124028+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:05.124028+0000","last_undegraded":"2021-01-15T18:47:05.124028+0000","last_fullsized":"2021-01-15T18:47:05.124028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7010,"ondisk_log_size":7010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22270035,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7009,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7025","reported_seq":"9973","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.028571+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:05.028571+0000","last_peered":"2021-01-15T18:47:05.028571+0000","last_clean":"2021-01-15T18:47:05.028571+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:05.028571+0000","last_undegraded":"2021-01-15T18:47:05.028571+0000","last_fullsized":"2021-01-15T18:47:05.028571+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7025,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22384889,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6816","reported_seq":"7064","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.150551+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:05.150551+0000","last_peered":"2021-01-15T18:47:05.150551+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:05.150551+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22429862,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6831","reported_seq":"9668","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.133023+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:05.133023+0000","last_peered":"2021-01-15T18:47:05.133023+0000","last_clean":"2021-01-15T18:47:05.133023+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:05.133023+0000","last_undegraded":"2021-01-15T18:47:05.133023+0000","last_fullsized":"2021-01-15T18:47:05.133023+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6831,"ondisk_log_size":6831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22413478,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6813","reported_seq":"9716","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:04.395644+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:04.395644+0000","last_peered":"2021-01-15T18:47:04.395644+0000","last_clean":"2021-01-15T18:47:04.395644+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:04.395644+0000","last_undegraded":"2021-01-15T18:47:04.395644+0000","last_fullsized":"2021-01-15T18:47:04.395644+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6813,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21807270,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7153","reported_seq":"7387","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.254839+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:05.254839+0000","last_peered":"2021-01-15T18:47:05.254839+0000","last_clean":"2021-01-15T18:47:05.254839+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:05.254839+0000","last_undegraded":"2021-01-15T18:47:05.254839+0000","last_fullsized":"2021-01-15T18:47:05.254839+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7153,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22388902,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7247","reported_seq":"7521","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.185079+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:05.185079+0000","last_peered":"2021-01-15T18:47:05.185079+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:05.185079+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7247,"ondisk_log_size":7247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22896972,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6853","reported_seq":"9726","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:03.576745+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:03.576745+0000","last_peered":"2021-01-15T18:47:03.576745+0000","last_clean":"2021-01-15T18:47:03.576745+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:03.576745+0000","last_undegraded":"2021-01-15T18:47:03.576745+0000","last_fullsized":"2021-01-15T18:47:03.576745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6853,"ondisk_log_size":6853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22003878,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":357034284,"num_objects":43598,"num_object_clones":0,"num_object_copies":174392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111127,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13854,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715034624,"data_stored":715034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111154,"ondisk_log_size":111154,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397200,"kb_used_data":174940,"kb_used_omap":0,"kb_used_meta":222207,"kb_avail":93974640,"statfs":{"total":96636764160,"available":96230031360,"internally_reserved":0,"allocated":179138560,"data_stored":178885382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":227540020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,4,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":3.0139999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":7.0819999999999999},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.6619999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398080,"kb_used_data":176316,"kb_used_omap":0,"kb_used_meta":221696,"kb_avail":93973760,"statfs":{"total":96636764160,"available":96229130240,"internally_reserved":0,"allocated":180547584,"data_stored":180294406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":227016704},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.0880000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":4.1980000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":3.2410000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.5519999999999996},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":4.3019999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393312,"kb_used_data":175320,"kb_used_omap":0,"kb_used_meta":217983,"kb_avail":93978528,"statfs":{"total":96636764160,"available":96234012672,"internally_reserved":0,"allocated":179527680,"data_stored":179274502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":223214644},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":8.7070000000000007},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":1.0229999999999999},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.1150000000000002},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":2.3340000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386168,"kb_used_data":175108,"kb_used_omap":0,"kb_used_meta":211007,"kb_avail":93985672,"statfs":{"total":96636764160,"available":96241328128,"internally_reserved":0,"allocated":179310592,"data_stored":179057414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":216071220},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":30.905999999999999},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":5.8140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.208}]}]}]}} 2021-01-15T18:47:07.829 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:08.961 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:08.962 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7553/174392 objects degraded (4.331%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":479,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43598,"data_bytes":357034284,"bytes_used":1612554240,"bytes_avail":384934502400,"bytes_total":386547056640,"degraded_objects":7553,"degraded_total":174392,"degraded_ratio":0.043310472957475116,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":179},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:08.988 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:08.989 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:10.200 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:10.288 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2021-01-15T18:47:08.585866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357034284,"num_objects":43598,"num_object_clones":0,"num_object_copies":174392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111127,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13854,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111154,"ondisk_log_size":111154,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1574760,"kb_used_data":701684,"kb_used_omap":2,"kb_used_meta":872893,"kb_avail":375912600,"statfs":{"total":386547056640,"available":384934502400,"internally_reserved":0,"allocated":718524416,"data_stored":717511704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":893842588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,4,7,13,22,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17678336,"num_objects":-2158,"num_object_clones":0,"num_object_copies":-8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006057"},"pg_stats":[{"pgid":"1.f","version":"48'6759","reported_seq":"7025","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.083414+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:05.083414+0000","last_peered":"2021-01-15T18:47:05.083414+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:05.083414+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6759,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22519808,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6759,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6809","reported_seq":"9741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.207087+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:05.207087+0000","last_peered":"2021-01-15T18:47:05.207087+0000","last_clean":"2021-01-15T18:47:05.207087+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:05.207087+0000","last_undegraded":"2021-01-15T18:47:05.207087+0000","last_fullsized":"2021-01-15T18:47:05.207087+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21725350,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6833","reported_seq":"9772","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:03.358234+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:03.358234+0000","last_peered":"2021-01-15T18:47:03.358234+0000","last_clean":"2021-01-15T18:47:03.358234+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:03.358234+0000","last_undegraded":"2021-01-15T18:47:03.358234+0000","last_fullsized":"2021-01-15T18:47:03.358234+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6833,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21778539,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6832,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7045","reported_seq":"7299","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:04.847105+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:04.847105+0000","last_peered":"2021-01-15T18:47:04.847105+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:04.847105+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7045,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22503424,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7097","reported_seq":"7697","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.274715+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:05.274715+0000","last_peered":"2021-01-15T18:47:05.274715+0000","last_clean":"2021-01-15T18:47:05.274715+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:05.274715+0000","last_undegraded":"2021-01-15T18:47:05.274715+0000","last_fullsized":"2021-01-15T18:47:05.274715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7097,"ondisk_log_size":7097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22372518,"num_objects":2732,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6820","reported_seq":"7039","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.185751+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:05.185751+0000","last_peered":"2021-01-15T18:47:05.185751+0000","last_clean":"2021-01-15T18:47:05.185751+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:05.185751+0000","last_undegraded":"2021-01-15T18:47:05.185751+0000","last_fullsized":"2021-01-15T18:47:05.185751+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6820,"ondisk_log_size":6820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22167718,"num_objects":2707,"num_object_clones":0,"num_object_copies":10828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6818,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'6965","reported_seq":"8074","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.246283+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:05.246283+0000","last_peered":"2021-01-15T18:47:05.246283+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:05.246283+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6965,"ondisk_log_size":6965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22610086,"num_objects":2761,"num_object_clones":0,"num_object_copies":11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7078","reported_seq":"7242","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:03.419982+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:03.419982+0000","last_peered":"2021-01-15T18:47:03.419982+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:03.419982+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7078,"ondisk_log_size":7078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22761555,"num_objects":2779,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7010","reported_seq":"9953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.124028+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:05.124028+0000","last_peered":"2021-01-15T18:47:05.124028+0000","last_clean":"2021-01-15T18:47:05.124028+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:05.124028+0000","last_undegraded":"2021-01-15T18:47:05.124028+0000","last_fullsized":"2021-01-15T18:47:05.124028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7010,"ondisk_log_size":7010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22270035,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7009,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7025","reported_seq":"9973","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.028571+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:05.028571+0000","last_peered":"2021-01-15T18:47:05.028571+0000","last_clean":"2021-01-15T18:47:05.028571+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:05.028571+0000","last_undegraded":"2021-01-15T18:47:05.028571+0000","last_fullsized":"2021-01-15T18:47:05.028571+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7025,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22384889,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6816","reported_seq":"7064","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.150551+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:05.150551+0000","last_peered":"2021-01-15T18:47:05.150551+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:05.150551+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22429862,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6831","reported_seq":"9668","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.133023+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:05.133023+0000","last_peered":"2021-01-15T18:47:05.133023+0000","last_clean":"2021-01-15T18:47:05.133023+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:05.133023+0000","last_undegraded":"2021-01-15T18:47:05.133023+0000","last_fullsized":"2021-01-15T18:47:05.133023+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6831,"ondisk_log_size":6831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22413478,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6829,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6813","reported_seq":"9716","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:04.395644+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:04.395644+0000","last_peered":"2021-01-15T18:47:04.395644+0000","last_clean":"2021-01-15T18:47:04.395644+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:04.395644+0000","last_undegraded":"2021-01-15T18:47:04.395644+0000","last_fullsized":"2021-01-15T18:47:04.395644+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6813,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21807270,"num_objects":2663,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7153","reported_seq":"7387","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:05.254839+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:05.254839+0000","last_peered":"2021-01-15T18:47:05.254839+0000","last_clean":"2021-01-15T18:47:05.254839+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:05.254839+0000","last_undegraded":"2021-01-15T18:47:05.254839+0000","last_fullsized":"2021-01-15T18:47:05.254839+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7153,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22388902,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7247","reported_seq":"7521","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:05.185079+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:05.185079+0000","last_peered":"2021-01-15T18:47:05.185079+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:05.185079+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7247,"ondisk_log_size":7247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22896972,"num_objects":2797,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6853","reported_seq":"9726","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:03.576745+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:03.576745+0000","last_peered":"2021-01-15T18:47:03.576745+0000","last_clean":"2021-01-15T18:47:03.576745+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:03.576745+0000","last_undegraded":"2021-01-15T18:47:03.576745+0000","last_fullsized":"2021-01-15T18:47:03.576745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6853,"ondisk_log_size":6853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22003878,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6851,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":357034284,"num_objects":43598,"num_object_clones":0,"num_object_copies":174392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111127,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13854,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715034624,"data_stored":715034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111154,"ondisk_log_size":111154,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215195,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397200,"kb_used_data":174940,"kb_used_omap":0,"kb_used_meta":222207,"kb_avail":93974640,"statfs":{"total":96636764160,"available":96230031360,"internally_reserved":0,"allocated":179138560,"data_stored":178885382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":227540020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,4,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":3.4910000000000001,"15min":3.169},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.8740000000000006,"5min":22.928000000000001,"15min":22.928000000000001},"last":3.0139999999999998},{"interface":"front","average":{"1min":1.423,"5min":3.1459999999999999,"15min":3.3599999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":10.125,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.89200000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:46:05 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.5329999999999999,"15min":2.5070000000000001},"min":{"1min":0.46999999999999997,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":24.062000000000001,"5min":24.062000000000001,"15min":24.062000000000001},"last":7.0819999999999999},{"interface":"front","average":{"1min":3.3929999999999998,"5min":4.2519999999999998,"15min":3.3149999999999999},"min":{"1min":0.60299999999999998,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":23.154,"5min":23.154,"15min":23.154},"last":1.6619999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":398080,"kb_used_data":176316,"kb_used_omap":0,"kb_used_meta":221696,"kb_avail":93973760,"statfs":{"total":96636764160,"available":96229130240,"internally_reserved":0,"allocated":180547584,"data_stored":180294406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":227016704},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.0880000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":4.1980000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":3.2410000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.5519999999999996},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":4.3019999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393312,"kb_used_data":175320,"kb_used_omap":0,"kb_used_meta":217983,"kb_avail":93978528,"statfs":{"total":96636764160,"available":96234012672,"internally_reserved":0,"allocated":179527680,"data_stored":179274502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":223214644},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":8.7070000000000007},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":1.0229999999999999},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.1150000000000002},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":2.3340000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386168,"kb_used_data":175108,"kb_used_omap":0,"kb_used_meta":211007,"kb_avail":93985672,"statfs":{"total":96636764160,"available":96241328128,"internally_reserved":0,"allocated":179310592,"data_stored":179057414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":216071220},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":30.905999999999999},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":5.8140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.208}]}]}]}} 2021-01-15T18:47:13.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:14.481 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:14.499 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:14.527 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2021-01-15T18:47:12.586763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348776748,"num_objects":42590,"num_object_clones":0,"num_object_copies":170360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112135,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13954,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112162,"ondisk_log_size":112162,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1533484,"kb_used_data":685660,"kb_used_omap":2,"kb_used_meta":847613,"kb_avail":375953876,"statfs":{"total":386547056640,"available":384976769024,"internally_reserved":0,"allocated":702115840,"data_stored":701103128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":867955868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,11,23,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":123,"apply_latency_ms":123,"commit_latency_ns":123000000,"apply_latency_ns":123000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16678912,"num_objects":-2036,"num_object_clones":0,"num_object_copies":-8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004254"},"pg_stats":[{"pgid":"1.f","version":"48'6828","reported_seq":"7092","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:10.211795+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:10.211795+0000","last_peered":"2021-01-15T18:47:10.211795+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:10.211795+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6828,"ondisk_log_size":6828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21954560,"num_objects":2680,"num_object_clones":0,"num_object_copies":10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6876","reported_seq":"9808","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:10.084133+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:10.084133+0000","last_peered":"2021-01-15T18:47:10.084133+0000","last_clean":"2021-01-15T18:47:10.084133+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:10.084133+0000","last_undegraded":"2021-01-15T18:47:10.084133+0000","last_fullsized":"2021-01-15T18:47:10.084133+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6876,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21176486,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6895","reported_seq":"9834","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:08.518095+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:08.518095+0000","last_peered":"2021-01-15T18:47:08.518095+0000","last_clean":"2021-01-15T18:47:08.518095+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:08.518095+0000","last_undegraded":"2021-01-15T18:47:08.518095+0000","last_fullsized":"2021-01-15T18:47:08.518095+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6895,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21270635,"num_objects":2598,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6894,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7116","reported_seq":"7369","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:10.258375+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:10.258375+0000","last_peered":"2021-01-15T18:47:10.258375+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:10.258375+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7116,"ondisk_log_size":7116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21921792,"num_objects":2676,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7169","reported_seq":"7769","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:10.175826+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:10.175826+0000","last_peered":"2021-01-15T18:47:10.175826+0000","last_clean":"2021-01-15T18:47:10.175826+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:10.175826+0000","last_undegraded":"2021-01-15T18:47:10.175826+0000","last_fullsized":"2021-01-15T18:47:10.175826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7169,"ondisk_log_size":7169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21782694,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6883","reported_seq":"7101","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:10.233078+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:10.233078+0000","last_peered":"2021-01-15T18:47:10.233078+0000","last_clean":"2021-01-15T18:47:10.233078+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:10.233078+0000","last_undegraded":"2021-01-15T18:47:10.233078+0000","last_fullsized":"2021-01-15T18:47:10.233078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6883,"ondisk_log_size":6883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21651622,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6881,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7025","reported_seq":"8334","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:10.231432+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:10.231432+0000","last_peered":"2021-01-15T18:47:10.231432+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:10.231432+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7025,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22118566,"num_objects":2701,"num_object_clones":0,"num_object_copies":10804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7137","reported_seq":"7301","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:08.567186+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:08.567186+0000","last_peered":"2021-01-15T18:47:08.567186+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:08.567186+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7137,"ondisk_log_size":7137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22278227,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7136,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7073","reported_seq":"10016","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:09.999463+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:09.999463+0000","last_peered":"2021-01-15T18:47:09.999463+0000","last_clean":"2021-01-15T18:47:09.999463+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:09.999463+0000","last_undegraded":"2021-01-15T18:47:09.999463+0000","last_fullsized":"2021-01-15T18:47:09.999463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7073,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21753939,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7072,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7099","reported_seq":"10048","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:10.204072+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:10.204072+0000","last_peered":"2021-01-15T18:47:10.204072+0000","last_clean":"2021-01-15T18:47:10.204072+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:10.204072+0000","last_undegraded":"2021-01-15T18:47:10.204072+0000","last_fullsized":"2021-01-15T18:47:10.204072+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7099,"ondisk_log_size":7099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21778681,"num_objects":2660,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6878","reported_seq":"7126","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:10.094601+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:10.094601+0000","last_peered":"2021-01-15T18:47:10.094601+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:10.094601+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6878,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21921958,"num_objects":2677,"num_object_clones":0,"num_object_copies":10708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6876,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1385}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6882","reported_seq":"9719","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:09.916679+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:09.916679+0000","last_peered":"2021-01-15T18:47:09.916679+0000","last_clean":"2021-01-15T18:47:09.916679+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:09.916679+0000","last_undegraded":"2021-01-15T18:47:09.916679+0000","last_fullsized":"2021-01-15T18:47:09.916679+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6882,"ondisk_log_size":6882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21995686,"num_objects":2686,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6859","reported_seq":"9762","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:09.247424+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:09.247424+0000","last_peered":"2021-01-15T18:47:09.247424+0000","last_clean":"2021-01-15T18:47:09.247424+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:09.247424+0000","last_undegraded":"2021-01-15T18:47:09.247424+0000","last_fullsized":"2021-01-15T18:47:09.247424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6859,"ondisk_log_size":6859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21430438,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6857,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7218","reported_seq":"7452","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:10.140038+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:10.140038+0000","last_peered":"2021-01-15T18:47:10.140038+0000","last_clean":"2021-01-15T18:47:10.140038+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:10.140038+0000","last_undegraded":"2021-01-15T18:47:10.140038+0000","last_fullsized":"2021-01-15T18:47:10.140038+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7218,"ondisk_log_size":7218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21856422,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7310","reported_seq":"7584","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:10.147928+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:10.147928+0000","last_peered":"2021-01-15T18:47:10.147928+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:10.147928+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7310,"ondisk_log_size":7310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22380876,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6914","reported_seq":"9787","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:08.507068+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:08.507068+0000","last_peered":"2021-01-15T18:47:08.507068+0000","last_clean":"2021-01-15T18:47:08.507068+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:08.507068+0000","last_undegraded":"2021-01-15T18:47:08.507068+0000","last_fullsized":"2021-01-15T18:47:08.507068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6914,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21504166,"num_objects":2626,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":348776748,"num_objects":42590,"num_object_clones":0,"num_object_copies":170360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42590,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112135,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13954,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":698462208,"data_stored":698462208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112162,"ondisk_log_size":112162,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215196,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386808,"kb_used_data":170944,"kb_used_omap":0,"kb_used_meta":215807,"kb_avail":93985032,"statfs":{"total":96636764160,"available":96240672768,"internally_reserved":0,"allocated":175046656,"data_stored":174793478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220986420},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.66500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.97299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":1.0760000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387776,"kb_used_data":172360,"kb_used_omap":0,"kb_used_meta":215360,"kb_avail":93984064,"statfs":{"total":96636764160,"available":96239681536,"internally_reserved":0,"allocated":176496640,"data_stored":176243462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220528640},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.925},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":0.57399999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382980,"kb_used_data":171296,"kb_used_omap":0,"kb_used_meta":211647,"kb_avail":93988860,"statfs":{"total":96636764160,"available":96244592640,"internally_reserved":0,"allocated":175407104,"data_stored":175153926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":216726580},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":19.954999999999998},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":5.335}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":4.952},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":9.5020000000000007},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":1.1000000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375920,"kb_used_data":171060,"kb_used_omap":0,"kb_used_meta":204799,"kb_avail":93995920,"statfs":{"total":96636764160,"available":96251822080,"internally_reserved":0,"allocated":175165440,"data_stored":174912262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":209714228},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.8980000000000001},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":5.6559999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":1.111},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":12.609999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.4339999999999999}]}]}]}} 2021-01-15T18:47:14.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:16.085 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:16.085 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7453/169320 objects degraded (4.402%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":486,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42330,"data_bytes":346646828,"bytes_used":1582186496,"bytes_avail":384964870144,"bytes_total":386547056640,"degraded_objects":7453,"degraded_total":169320,"degraded_ratio":0.044017245452397817,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:16.164 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:16.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:17.406 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:17.458 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2021-01-15T18:47:16.587640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":339814700,"num_objects":41496,"num_object_clones":0,"num_object_copies":165984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113229,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14012,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113256,"ondisk_log_size":113256,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1556840,"kb_used_data":668084,"kb_used_omap":2,"kb_used_meta":888637,"kb_avail":375930520,"statfs":{"total":386547056640,"available":384952852480,"internally_reserved":0,"allocated":684118016,"data_stored":683105304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":909964444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,8,9,21,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23592960,"num_objects":-2880,"num_object_clones":0,"num_object_copies":-11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004233"},"pg_stats":[{"pgid":"1.f","version":"48'6908","reported_seq":"7172","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:15.262457+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:15.262457+0000","last_peered":"2021-01-15T18:47:15.262457+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:15.262457+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6908,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21299200,"num_objects":2600,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'6949","reported_seq":"9880","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.268917+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:15.268917+0000","last_peered":"2021-01-15T18:47:15.268917+0000","last_clean":"2021-01-15T18:47:15.268917+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:15.268917+0000","last_undegraded":"2021-01-15T18:47:15.268917+0000","last_fullsized":"2021-01-15T18:47:15.268917+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6949,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20578470,"num_objects":2513,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'6953","reported_seq":"9890","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:13.482161+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:13.482161+0000","last_peered":"2021-01-15T18:47:13.482161+0000","last_clean":"2021-01-15T18:47:13.482161+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:13.482161+0000","last_undegraded":"2021-01-15T18:47:13.482161+0000","last_fullsized":"2021-01-15T18:47:13.482161+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6953,"ondisk_log_size":6953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20795499,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6952,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7184","reported_seq":"7438","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:15.114032+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:15.114032+0000","last_peered":"2021-01-15T18:47:15.114032+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:15.114032+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7184,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21364736,"num_objects":2608,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7237","reported_seq":"7837","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.228558+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:15.228558+0000","last_peered":"2021-01-15T18:47:15.228558+0000","last_clean":"2021-01-15T18:47:15.228558+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:15.228558+0000","last_undegraded":"2021-01-15T18:47:15.228558+0000","last_fullsized":"2021-01-15T18:47:15.228558+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7237,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21225638,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7235,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'6951","reported_seq":"7170","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.173314+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:15.173314+0000","last_peered":"2021-01-15T18:47:15.173314+0000","last_clean":"2021-01-15T18:47:15.173314+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:15.173314+0000","last_undegraded":"2021-01-15T18:47:15.173314+0000","last_fullsized":"2021-01-15T18:47:15.173314+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6951,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21094566,"num_objects":2576,"num_object_clones":0,"num_object_copies":10304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7081","reported_seq":"8506","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:15.227799+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:15.227799+0000","last_peered":"2021-01-15T18:47:15.227799+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:15.227799+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7081,"ondisk_log_size":7081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21659814,"num_objects":2645,"num_object_clones":0,"num_object_copies":10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7207","reported_seq":"7368","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:13.536630+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:13.536630+0000","last_peered":"2021-01-15T18:47:13.536630+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:13.536630+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7207,"ondisk_log_size":7207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21704787,"num_objects":2650,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7206,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1237}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7136","reported_seq":"10078","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.104637+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:15.104637+0000","last_peered":"2021-01-15T18:47:15.104637+0000","last_clean":"2021-01-15T18:47:15.104637+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:15.104637+0000","last_undegraded":"2021-01-15T18:47:15.104637+0000","last_fullsized":"2021-01-15T18:47:15.104637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7136,"ondisk_log_size":7136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21237843,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7135,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7167","reported_seq":"10116","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.192433+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:15.192433+0000","last_peered":"2021-01-15T18:47:15.192433+0000","last_clean":"2021-01-15T18:47:15.192433+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:15.192433+0000","last_undegraded":"2021-01-15T18:47:15.192433+0000","last_fullsized":"2021-01-15T18:47:15.192433+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7167,"ondisk_log_size":7167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21221625,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'6951","reported_seq":"7199","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:15.002726+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:15.002726+0000","last_peered":"2021-01-15T18:47:15.002726+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:15.002726+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6951,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21323942,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1312}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'6963","reported_seq":"9800","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.017304+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:15.017304+0000","last_peered":"2021-01-15T18:47:15.017304+0000","last_clean":"2021-01-15T18:47:15.017304+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:15.017304+0000","last_undegraded":"2021-01-15T18:47:15.017304+0000","last_fullsized":"2021-01-15T18:47:15.017304+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6963,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21332134,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6932","reported_seq":"9835","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:14.450022+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:14.450022+0000","last_peered":"2021-01-15T18:47:14.450022+0000","last_clean":"2021-01-15T18:47:14.450022+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:14.450022+0000","last_undegraded":"2021-01-15T18:47:14.450022+0000","last_fullsized":"2021-01-15T18:47:14.450022+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6932,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20832422,"num_objects":2544,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7276","reported_seq":"7510","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:15.272719+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:15.272719+0000","last_peered":"2021-01-15T18:47:15.272719+0000","last_clean":"2021-01-15T18:47:15.272719+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:15.272719+0000","last_undegraded":"2021-01-15T18:47:15.272719+0000","last_fullsized":"2021-01-15T18:47:15.272719+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7276,"ondisk_log_size":7276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21381286,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7388","reported_seq":"7661","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:15.114264+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:15.114264+0000","last_peered":"2021-01-15T18:47:15.114264+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:15.114264+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7388,"ondisk_log_size":7388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21741900,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7385,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1271}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'6973","reported_seq":"9846","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:13.710653+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:13.710653+0000","last_peered":"2021-01-15T18:47:13.710653+0000","last_clean":"2021-01-15T18:47:13.710653+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:13.710653+0000","last_undegraded":"2021-01-15T18:47:13.710653+0000","last_fullsized":"2021-01-15T18:47:13.710653+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6973,"ondisk_log_size":6973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21020838,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6971,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":339814700,"num_objects":41496,"num_object_clones":0,"num_object_copies":165984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113229,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14012,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":680591360,"data_stored":680591360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113256,"ondisk_log_size":113256,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215197,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392676,"kb_used_data":166568,"kb_used_omap":0,"kb_used_meta":226047,"kb_avail":93979164,"statfs":{"total":96636764160,"available":96234663936,"internally_reserved":0,"allocated":170565632,"data_stored":170312454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":231472180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,0,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":1.796},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.1269999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.8979999999999999},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.0209999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":2.246}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393576,"kb_used_data":167960,"kb_used_omap":0,"kb_used_meta":225600,"kb_avail":93978264,"statfs":{"total":96636764160,"available":96233742336,"internally_reserved":0,"allocated":171991040,"data_stored":171737862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":231014400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.1679999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":4.8010000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":1.04}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.4059999999999997},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.4870000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388800,"kb_used_data":166892,"kb_used_omap":0,"kb_used_meta":221887,"kb_avail":93983040,"statfs":{"total":96636764160,"available":96238632960,"internally_reserved":0,"allocated":170897408,"data_stored":170644230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":227212340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,0,0,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":4.5110000000000001},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":4.6239999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":4.6020000000000003},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":7.4160000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":7.7859999999999996},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":7.5529999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381788,"kb_used_data":166664,"kb_used_omap":0,"kb_used_meta":215103,"kb_avail":93990052,"statfs":{"total":96636764160,"available":96245813248,"internally_reserved":0,"allocated":170663936,"data_stored":170410758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220265524},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.7909999999999999},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":2.7879999999999998},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.071}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":3.3279999999999998},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":3.4369999999999998}]}]}]}} 2021-01-15T18:47:20.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:21.654 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:21.687 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:21.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2021-01-15T18:47:20.588554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330942764,"num_objects":40413,"num_object_clones":0,"num_object_copies":161652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40413,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114312,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14061,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114339,"ondisk_log_size":114339,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1510380,"kb_used_data":650240,"kb_used_omap":2,"kb_used_meta":859965,"kb_avail":375976980,"statfs":{"total":386547056640,"available":385000427520,"internally_reserved":0,"allocated":665845760,"data_stored":664833048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":880604316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,9,17,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":121,"apply_latency_ms":121,"commit_latency_ns":121000000,"apply_latency_ns":121000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26091520,"num_objects":-3185,"num_object_clones":0,"num_object_copies":-12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002688"},"pg_stats":[{"pgid":"1.f","version":"48'6960","reported_seq":"7225","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:19.982509+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:19.982509+0000","last_peered":"2021-01-15T18:47:19.982509+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:19.982509+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6960,"ondisk_log_size":6960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20873216,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7024","reported_seq":"9953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.065889+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:20.065889+0000","last_peered":"2021-01-15T18:47:20.065889+0000","last_clean":"2021-01-15T18:47:20.065889+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:20.065889+0000","last_undegraded":"2021-01-15T18:47:20.065889+0000","last_fullsized":"2021-01-15T18:47:20.065889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7024,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7024","reported_seq":"9963","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:18.643549+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:18.643549+0000","last_peered":"2021-01-15T18:47:18.643549+0000","last_clean":"2021-01-15T18:47:18.643549+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:18.643549+0000","last_undegraded":"2021-01-15T18:47:18.643549+0000","last_fullsized":"2021-01-15T18:47:18.643549+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7024,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20213867,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7023,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7259","reported_seq":"7513","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.033513+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:20.033513+0000","last_peered":"2021-01-15T18:47:20.033513+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:20.033513+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7259,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7306","reported_seq":"7906","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.886363+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:19.886363+0000","last_peered":"2021-01-15T18:47:19.886363+0000","last_clean":"2021-01-15T18:47:19.886363+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:19.886363+0000","last_undegraded":"2021-01-15T18:47:19.886363+0000","last_fullsized":"2021-01-15T18:47:19.886363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7306,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20660390,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7018","reported_seq":"7237","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.585362+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:19.585362+0000","last_peered":"2021-01-15T18:47:19.585362+0000","last_clean":"2021-01-15T18:47:19.585362+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:19.585362+0000","last_undegraded":"2021-01-15T18:47:19.585362+0000","last_fullsized":"2021-01-15T18:47:19.585362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7018,"ondisk_log_size":7018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20545702,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7149","reported_seq":"8671","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.048587+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:20.048587+0000","last_peered":"2021-01-15T18:47:20.048587+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:20.048587+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7149,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21102758,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7271","reported_seq":"7434","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:18.503040+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:18.503040+0000","last_peered":"2021-01-15T18:47:18.503040+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:18.503040+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7271,"ondisk_log_size":7271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21180499,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7208","reported_seq":"10151","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.961481+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:19.961481+0000","last_peered":"2021-01-15T18:47:19.961481+0000","last_clean":"2021-01-15T18:47:19.961481+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:19.961481+0000","last_undegraded":"2021-01-15T18:47:19.961481+0000","last_fullsized":"2021-01-15T18:47:19.961481+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7208,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20648019,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7236","reported_seq":"10184","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.278203+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:20.278203+0000","last_peered":"2021-01-15T18:47:20.278203+0000","last_clean":"2021-01-15T18:47:20.278203+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:20.278203+0000","last_undegraded":"2021-01-15T18:47:20.278203+0000","last_fullsized":"2021-01-15T18:47:20.278203+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7236,"ondisk_log_size":7236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20656377,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7007","reported_seq":"7255","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.279016+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:20.279016+0000","last_peered":"2021-01-15T18:47:20.279016+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:20.279016+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7007,"ondisk_log_size":7007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20865190,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7020","reported_seq":"9857","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.061764+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:20.061764+0000","last_peered":"2021-01-15T18:47:20.061764+0000","last_clean":"2021-01-15T18:47:20.061764+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:20.061764+0000","last_undegraded":"2021-01-15T18:47:20.061764+0000","last_fullsized":"2021-01-15T18:47:20.061764+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7020,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20865190,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'6994","reported_seq":"9897","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.378063+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:19.378063+0000","last_peered":"2021-01-15T18:47:19.378063+0000","last_clean":"2021-01-15T18:47:19.378063+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:19.378063+0000","last_undegraded":"2021-01-15T18:47:19.378063+0000","last_fullsized":"2021-01-15T18:47:19.378063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6994,"ondisk_log_size":6994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324518,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7352","reported_seq":"7584","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.277899+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:20.277899+0000","last_peered":"2021-01-15T18:47:20.277899+0000","last_clean":"2021-01-15T18:47:20.277899+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:20.277899+0000","last_undegraded":"2021-01-15T18:47:20.277899+0000","last_fullsized":"2021-01-15T18:47:20.277899+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7352,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20758694,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7463","reported_seq":"7737","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:19.971110+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:19.971110+0000","last_peered":"2021-01-15T18:47:19.971110+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:19.971110+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7463,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21127500,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7048","reported_seq":"9921","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:18.611493+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:18.611493+0000","last_peered":"2021-01-15T18:47:18.611493+0000","last_clean":"2021-01-15T18:47:18.611493+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:18.611493+0000","last_undegraded":"2021-01-15T18:47:18.611493+0000","last_fullsized":"2021-01-15T18:47:18.611493+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7048,"ondisk_log_size":7048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20406438,"num_objects":2492,"num_object_clones":0,"num_object_copies":9968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":330942764,"num_objects":40413,"num_object_clones":0,"num_object_copies":161652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40413,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114312,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14061,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662716416,"data_stored":662716416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114339,"ondisk_log_size":114339,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377900,"kb_used_data":162056,"kb_used_omap":0,"kb_used_meta":215807,"kb_avail":93993940,"statfs":{"total":96636764160,"available":96249794560,"internally_reserved":0,"allocated":165945344,"data_stored":165692166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220986420},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":3.3439999999999999},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":11.148},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":16.010000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":3.4929999999999999},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":3.2349999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383044,"kb_used_data":163596,"kb_used_omap":0,"kb_used_meta":219392,"kb_avail":93988796,"statfs":{"total":96636764160,"available":96244527104,"internally_reserved":0,"allocated":167522304,"data_stored":167269126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":224657408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":27.861999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.431},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":0.82199999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.4059999999999997},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.4870000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378144,"kb_used_data":162360,"kb_used_omap":0,"kb_used_meta":215743,"kb_avail":93993696,"statfs":{"total":96636764160,"available":96249544704,"internally_reserved":0,"allocated":166256640,"data_stored":166003462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220920884},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":21.806999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":0.68200000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":21.721},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.8959999999999999},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":2.0059999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371292,"kb_used_data":162228,"kb_used_omap":0,"kb_used_meta":209023,"kb_avail":94000548,"statfs":{"total":96636764160,"available":96256561152,"internally_reserved":0,"allocated":166121472,"data_stored":165868294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":214039604},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":17.263999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":2.0649999999999999},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":3.2839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":17.382999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":17.478000000000002}]}]}]}} 2021-01-15T18:47:21.722 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:23.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:23.282 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7346/161652 objects degraded (4.544%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40413,"data_bytes":330942764,"bytes_used":1546629120,"bytes_avail":385000427520,"bytes_total":386547056640,"degraded_objects":7346,"degraded_total":161652,"degraded_ratio":0.045443297948679878,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:23.379 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:23.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:24.779 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:24.813 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:24.828 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2021-01-15T18:47:24.591233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328861996,"num_objects":40159,"num_object_clones":0,"num_object_copies":160636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114566,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14061,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114593,"ondisk_log_size":114593,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1522520,"kb_used_data":642396,"kb_used_omap":2,"kb_used_meta":879997,"kb_avail":375964840,"statfs":{"total":386547056640,"available":384987996160,"internally_reserved":0,"allocated":657813504,"data_stored":656800792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":901117084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,9,14,15,17,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004470"},"pg_stats":[{"pgid":"1.f","version":"48'6960","reported_seq":"7225","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:19.982509+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:19.982509+0000","last_peered":"2021-01-15T18:47:19.982509+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:19.982509+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":6960,"ondisk_log_size":6960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20873216,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7024","reported_seq":"9953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.065889+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:20.065889+0000","last_peered":"2021-01-15T18:47:20.065889+0000","last_clean":"2021-01-15T18:47:20.065889+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:20.065889+0000","last_undegraded":"2021-01-15T18:47:20.065889+0000","last_fullsized":"2021-01-15T18:47:20.065889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7024,"ondisk_log_size":7024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19964070,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7091","reported_seq":"10030","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:23.683358+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:23.683358+0000","last_peered":"2021-01-15T18:47:23.683358+0000","last_clean":"2021-01-15T18:47:23.683358+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:23.683358+0000","last_undegraded":"2021-01-15T18:47:23.683358+0000","last_fullsized":"2021-01-15T18:47:23.683358+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7091,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19665003,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7090,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7259","reported_seq":"7513","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.033513+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:20.033513+0000","last_peered":"2021-01-15T18:47:20.033513+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:20.033513+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7259,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20750336,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7306","reported_seq":"7906","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.886363+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:19.886363+0000","last_peered":"2021-01-15T18:47:19.886363+0000","last_clean":"2021-01-15T18:47:19.886363+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:19.886363+0000","last_undegraded":"2021-01-15T18:47:19.886363+0000","last_fullsized":"2021-01-15T18:47:19.886363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7306,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20660390,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7018","reported_seq":"7237","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.585362+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:19.585362+0000","last_peered":"2021-01-15T18:47:19.585362+0000","last_clean":"2021-01-15T18:47:19.585362+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:19.585362+0000","last_undegraded":"2021-01-15T18:47:19.585362+0000","last_fullsized":"2021-01-15T18:47:19.585362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7018,"ondisk_log_size":7018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20545702,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7149","reported_seq":"8671","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.048587+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:20.048587+0000","last_peered":"2021-01-15T18:47:20.048587+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:20.048587+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7149,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21102758,"num_objects":2577,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7331","reported_seq":"7495","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:23.508689+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:23.508689+0000","last_peered":"2021-01-15T18:47:23.508689+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:23.508689+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7331,"ondisk_log_size":7331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20688979,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1113}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7208","reported_seq":"10151","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:19.961481+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:19.961481+0000","last_peered":"2021-01-15T18:47:19.961481+0000","last_clean":"2021-01-15T18:47:19.961481+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:19.961481+0000","last_undegraded":"2021-01-15T18:47:19.961481+0000","last_fullsized":"2021-01-15T18:47:19.961481+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7208,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20648019,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7236","reported_seq":"10184","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.278203+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:20.278203+0000","last_peered":"2021-01-15T18:47:20.278203+0000","last_clean":"2021-01-15T18:47:20.278203+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:20.278203+0000","last_undegraded":"2021-01-15T18:47:20.278203+0000","last_fullsized":"2021-01-15T18:47:20.278203+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7236,"ondisk_log_size":7236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20656377,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7007","reported_seq":"7255","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:20.279016+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:20.279016+0000","last_peered":"2021-01-15T18:47:20.279016+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:20.279016+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7007,"ondisk_log_size":7007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20865190,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7020","reported_seq":"9857","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.061764+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:20.061764+0000","last_peered":"2021-01-15T18:47:20.061764+0000","last_clean":"2021-01-15T18:47:20.061764+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:20.061764+0000","last_undegraded":"2021-01-15T18:47:20.061764+0000","last_fullsized":"2021-01-15T18:47:20.061764+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7020,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20865190,"num_objects":2548,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7057","reported_seq":"9960","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:24.425356+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:24.425356+0000","last_peered":"2021-01-15T18:47:24.425356+0000","last_clean":"2021-01-15T18:47:24.425356+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:24.425356+0000","last_undegraded":"2021-01-15T18:47:24.425356+0000","last_fullsized":"2021-01-15T18:47:24.425356+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7057,"ondisk_log_size":7057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19808422,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7352","reported_seq":"7584","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:20.277899+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:20.277899+0000","last_peered":"2021-01-15T18:47:20.277899+0000","last_clean":"2021-01-15T18:47:20.277899+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:20.277899+0000","last_undegraded":"2021-01-15T18:47:20.277899+0000","last_fullsized":"2021-01-15T18:47:20.277899+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7352,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20758694,"num_objects":2535,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7463","reported_seq":"7737","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:19.971110+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:19.971110+0000","last_peered":"2021-01-15T18:47:19.971110+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:19.971110+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7463,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21127500,"num_objects":2581,"num_object_clones":0,"num_object_copies":10324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7112","reported_seq":"9984","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:23.544181+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:23.544181+0000","last_peered":"2021-01-15T18:47:23.544181+0000","last_clean":"2021-01-15T18:47:23.544181+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:23.544181+0000","last_undegraded":"2021-01-15T18:47:23.544181+0000","last_fullsized":"2021-01-15T18:47:23.544181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7112,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328861996,"num_objects":40159,"num_object_clones":0,"num_object_copies":160636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114566,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14061,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654032896,"data_stored":654032896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114593,"ondisk_log_size":114593,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215198,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377900,"kb_used_data":162056,"kb_used_omap":0,"kb_used_meta":215807,"kb_avail":93993940,"statfs":{"total":96636764160,"available":96249794560,"internally_reserved":0,"allocated":165945344,"data_stored":165692166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220986420},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":3.3439999999999999},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":11.148},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":16.010000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":3.4929999999999999},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":3.2349999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388880,"kb_used_data":159280,"kb_used_omap":0,"kb_used_meta":229568,"kb_avail":93982960,"statfs":{"total":96636764160,"available":96238551040,"internally_reserved":0,"allocated":163102720,"data_stored":162849542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":235077632},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.3360000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":1.756}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.248},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":2.0289999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384448,"kb_used_data":158832,"kb_used_omap":0,"kb_used_meta":225599,"kb_avail":93987392,"statfs":{"total":96636764160,"available":96243089408,"internally_reserved":0,"allocated":162643968,"data_stored":162390790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":231013428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":2.7109999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":9.3209999999999997},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.92400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.82399999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371292,"kb_used_data":162228,"kb_used_omap":0,"kb_used_meta":209023,"kb_avail":94000548,"statfs":{"total":96636764160,"available":96256561152,"internally_reserved":0,"allocated":166121472,"data_stored":165868294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":214039604},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":17.263999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":2.0649999999999999},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":3.2839999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":17.382999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":17.478000000000002}]}]}]}} 2021-01-15T18:47:27.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:29.436 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:29.479 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2021-01-15T18:47:28.592155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322144556,"num_objects":39339,"num_object_clones":0,"num_object_copies":157356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115386,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14114,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115413,"ondisk_log_size":115413,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1530276,"kb_used_data":633824,"kb_used_omap":2,"kb_used_meta":896317,"kb_avail":375957084,"statfs":{"total":386547056640,"available":384980054016,"internally_reserved":0,"allocated":649035776,"data_stored":648023064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":917828764},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,7,23,20,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17670144,"num_objects":-2157,"num_object_clones":0,"num_object_copies":-8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"1.f","version":"48'7027","reported_seq":"7292","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:25.191829+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:25.191829+0000","last_peered":"2021-01-15T18:47:25.191829+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:25.191829+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7027,"ondisk_log_size":7027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20324352,"num_objects":2481,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7027,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7100","reported_seq":"10031","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.203435+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:25.203435+0000","last_peered":"2021-01-15T18:47:25.203435+0000","last_clean":"2021-01-15T18:47:25.203435+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:25.203435+0000","last_undegraded":"2021-01-15T18:47:25.203435+0000","last_fullsized":"2021-01-15T18:47:25.203435+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7100,"ondisk_log_size":7100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19341478,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7098,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7091","reported_seq":"10030","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:23.683358+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:23.683358+0000","last_peered":"2021-01-15T18:47:23.683358+0000","last_clean":"2021-01-15T18:47:23.683358+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:23.683358+0000","last_undegraded":"2021-01-15T18:47:23.683358+0000","last_fullsized":"2021-01-15T18:47:23.683358+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7091,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19665003,"num_objects":2402,"num_object_clones":0,"num_object_copies":9608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7090,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7326","reported_seq":"7580","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:25.096938+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:25.096938+0000","last_peered":"2021-01-15T18:47:25.096938+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:25.096938+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7326,"ondisk_log_size":7326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20201472,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7360","reported_seq":"7960","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.189046+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:25.189046+0000","last_peered":"2021-01-15T18:47:25.189046+0000","last_clean":"2021-01-15T18:47:25.189046+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:25.189046+0000","last_undegraded":"2021-01-15T18:47:25.189046+0000","last_fullsized":"2021-01-15T18:47:25.189046+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7360,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20218022,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7105","reported_seq":"7324","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.093265+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:25.093265+0000","last_peered":"2021-01-15T18:47:25.093265+0000","last_clean":"2021-01-15T18:47:25.093265+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:25.093265+0000","last_undegraded":"2021-01-15T18:47:25.093265+0000","last_fullsized":"2021-01-15T18:47:25.093265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7105,"ondisk_log_size":7105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19832998,"num_objects":2422,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7103,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7206","reported_seq":"8835","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:25.284447+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:25.284447+0000","last_peered":"2021-01-15T18:47:25.284447+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:25.284447+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7206,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20635814,"num_objects":2520,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7331","reported_seq":"7495","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:23.508689+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:23.508689+0000","last_peered":"2021-01-15T18:47:23.508689+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:23.508689+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7331,"ondisk_log_size":7331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20688979,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1113}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7284","reported_seq":"10227","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.204168+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:25.204168+0000","last_peered":"2021-01-15T18:47:25.204168+0000","last_clean":"2021-01-15T18:47:25.204168+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:25.204168+0000","last_undegraded":"2021-01-15T18:47:25.204168+0000","last_fullsized":"2021-01-15T18:47:25.204168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7284,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20025427,"num_objects":2445,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7283,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7314","reported_seq":"10262","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.169866+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:25.169866+0000","last_peered":"2021-01-15T18:47:25.169866+0000","last_clean":"2021-01-15T18:47:25.169866+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:25.169866+0000","last_undegraded":"2021-01-15T18:47:25.169866+0000","last_fullsized":"2021-01-15T18:47:25.169866+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7314,"ondisk_log_size":7314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20017401,"num_objects":2445,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7063","reported_seq":"7311","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:25.128271+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:25.128271+0000","last_peered":"2021-01-15T18:47:25.128271+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:25.128271+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7063,"ondisk_log_size":7063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20406438,"num_objects":2492,"num_object_clones":0,"num_object_copies":9968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7061,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1200}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7094","reported_seq":"9928","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.288499+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:25.288499+0000","last_peered":"2021-01-15T18:47:25.288499+0000","last_clean":"2021-01-15T18:47:25.288499+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:25.288499+0000","last_undegraded":"2021-01-15T18:47:25.288499+0000","last_fullsized":"2021-01-15T18:47:25.288499+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7094,"ondisk_log_size":7094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20258982,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7092,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7057","reported_seq":"9960","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:24.425356+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:24.425356+0000","last_peered":"2021-01-15T18:47:24.425356+0000","last_clean":"2021-01-15T18:47:24.425356+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:24.425356+0000","last_undegraded":"2021-01-15T18:47:24.425356+0000","last_fullsized":"2021-01-15T18:47:24.425356+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7057,"ondisk_log_size":7057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19808422,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7414","reported_seq":"7648","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:25.099774+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:25.099774+0000","last_peered":"2021-01-15T18:47:25.099774+0000","last_clean":"2021-01-15T18:47:25.099774+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:25.099774+0000","last_undegraded":"2021-01-15T18:47:25.099774+0000","last_fullsized":"2021-01-15T18:47:25.099774+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7414,"ondisk_log_size":7414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20250790,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7529","reported_seq":"7803","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:25.171230+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:25.171230+0000","last_peered":"2021-01-15T18:47:25.171230+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:25.171230+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7529,"ondisk_log_size":7529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20586828,"num_objects":2515,"num_object_clones":0,"num_object_copies":10060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7526,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7112","reported_seq":"9984","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:23.544181+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:23.544181+0000","last_peered":"2021-01-15T18:47:23.544181+0000","last_clean":"2021-01-15T18:47:23.544181+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:23.544181+0000","last_undegraded":"2021-01-15T18:47:23.544181+0000","last_fullsized":"2021-01-15T18:47:23.544181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7112,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19882150,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":322144556,"num_objects":39339,"num_object_clones":0,"num_object_copies":157356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115386,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14114,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":645017600,"data_stored":645017600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115413,"ondisk_log_size":115413,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215199,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383732,"kb_used_data":157640,"kb_used_omap":0,"kb_used_meta":226047,"kb_avail":93988108,"statfs":{"total":96636764160,"available":96243822592,"internally_reserved":0,"allocated":161423360,"data_stored":161170182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":231472180},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.76100000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.9830000000000001},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.077}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":26.803999999999998},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":1.7390000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388880,"kb_used_data":159280,"kb_used_omap":0,"kb_used_meta":229568,"kb_avail":93982960,"statfs":{"total":96636764160,"available":96238551040,"internally_reserved":0,"allocated":163102720,"data_stored":162849542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":235077632},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.3360000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":1.756}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.248},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":2.0289999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384448,"kb_used_data":158832,"kb_used_omap":0,"kb_used_meta":225599,"kb_avail":93987392,"statfs":{"total":96636764160,"available":96243089408,"internally_reserved":0,"allocated":162643968,"data_stored":162390790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":231013428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":2.7109999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":9.3209999999999997},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.92400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.82399999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373216,"kb_used_data":158072,"kb_used_omap":0,"kb_used_meta":215103,"kb_avail":93998624,"statfs":{"total":96636764160,"available":96254590976,"internally_reserved":0,"allocated":161865728,"data_stored":161612550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":220265524},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,8,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":3.2210000000000001},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.98399999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":2.7890000000000001},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":1.1819999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":1.077}]}]}]}} 2021-01-15T18:47:29.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:30.955 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:30.956 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7247/152744 objects degraded (4.745%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":501,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38186,"data_bytes":312699180,"bytes_used":1523048448,"bytes_avail":385024008192,"bytes_total":386547056640,"degraded_objects":7247,"degraded_total":152744,"degraded_ratio":0.047445398837270197,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":275},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:31.033 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:31.034 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:31.796 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:31.825 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:31.865 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2021-01-15T18:47:30.592598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":312699180,"num_objects":38186,"num_object_clones":0,"num_object_copies":152744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116539,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116566,"ondisk_log_size":116566,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1487352,"kb_used_data":614740,"kb_used_omap":2,"kb_used_meta":872445,"kb_avail":376000008,"statfs":{"total":386547056640,"available":385024008192,"internally_reserved":0,"allocated":629493760,"data_stored":628481048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":893383836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,3,12,15,8,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":126,"apply_latency_ms":126,"commit_latency_ns":126000000,"apply_latency_ns":126000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27115520,"num_objects":-3310,"num_object_clones":0,"num_object_copies":-13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004469"},"pg_stats":[{"pgid":"1.f","version":"48'7101","reported_seq":"7366","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.243879+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:30.243879+0000","last_peered":"2021-01-15T18:47:30.243879+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:30.243879+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7101,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7158","reported_seq":"10090","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.256693+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:30.256693+0000","last_peered":"2021-01-15T18:47:30.256693+0000","last_clean":"2021-01-15T18:47:30.256693+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:30.256693+0000","last_undegraded":"2021-01-15T18:47:30.256693+0000","last_fullsized":"2021-01-15T18:47:30.256693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7158,"ondisk_log_size":7158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18866342,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7156,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7181","reported_seq":"10120","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:28.548340+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:28.548340+0000","last_peered":"2021-01-15T18:47:28.548340+0000","last_clean":"2021-01-15T18:47:28.548340+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:28.548340+0000","last_undegraded":"2021-01-15T18:47:28.548340+0000","last_fullsized":"2021-01-15T18:47:28.548340+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7181,"ondisk_log_size":7181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18927723,"num_objects":2312,"num_object_clones":0,"num_object_copies":9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7180,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7392","reported_seq":"7646","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.148951+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:30.148951+0000","last_peered":"2021-01-15T18:47:30.148951+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:30.148951+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7392,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19660800,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7431","reported_seq":"8031","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.239106+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:30.239106+0000","last_peered":"2021-01-15T18:47:30.239106+0000","last_clean":"2021-01-15T18:47:30.239106+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:30.239106+0000","last_undegraded":"2021-01-15T18:47:30.239106+0000","last_fullsized":"2021-01-15T18:47:30.239106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7431,"ondisk_log_size":7431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19636390,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7429,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7179","reported_seq":"7398","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.343168+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:30.343168+0000","last_peered":"2021-01-15T18:47:30.343168+0000","last_clean":"2021-01-15T18:47:30.343168+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:30.343168+0000","last_undegraded":"2021-01-15T18:47:30.343168+0000","last_fullsized":"2021-01-15T18:47:30.343168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7179,"ondisk_log_size":7179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19226790,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7272","reported_seq":"8993","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.364423+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:30.364423+0000","last_peered":"2021-01-15T18:47:30.364423+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:30.364423+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7272,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20095142,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7401","reported_seq":"7565","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:28.738251+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:28.738251+0000","last_peered":"2021-01-15T18:47:28.738251+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:28.738251+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7401,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20115539,"num_objects":2456,"num_object_clones":0,"num_object_copies":9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7400,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7357","reported_seq":"10300","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.161779+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:30.161779+0000","last_peered":"2021-01-15T18:47:30.161779+0000","last_clean":"2021-01-15T18:47:30.161779+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:30.161779+0000","last_undegraded":"2021-01-15T18:47:30.161779+0000","last_fullsized":"2021-01-15T18:47:30.161779+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7357,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19427411,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7377","reported_seq":"10326","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.280882+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:30.280882+0000","last_peered":"2021-01-15T18:47:30.280882+0000","last_clean":"2021-01-15T18:47:30.280882+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:30.280882+0000","last_undegraded":"2021-01-15T18:47:30.280882+0000","last_fullsized":"2021-01-15T18:47:30.280882+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7377,"ondisk_log_size":7377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19501305,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7145","reported_seq":"7393","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.275811+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:30.275811+0000","last_peered":"2021-01-15T18:47:30.275811+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:30.275811+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7145,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19734694,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7153","reported_seq":"9990","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.350646+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:30.350646+0000","last_peered":"2021-01-15T18:47:30.350646+0000","last_clean":"2021-01-15T18:47:30.350646+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:30.350646+0000","last_undegraded":"2021-01-15T18:47:30.350646+0000","last_fullsized":"2021-01-15T18:47:30.350646+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7153,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19775654,"num_objects":2415,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7130","reported_seq":"10032","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:29.506102+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:29.506102+0000","last_peered":"2021-01-15T18:47:29.506102+0000","last_clean":"2021-01-15T18:47:29.506102+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:29.506102+0000","last_undegraded":"2021-01-15T18:47:29.506102+0000","last_fullsized":"2021-01-15T18:47:29.506102+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7130,"ondisk_log_size":7130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19210406,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7128,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7491","reported_seq":"7724","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.346017+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:30.346017+0000","last_peered":"2021-01-15T18:47:30.346017+0000","last_clean":"2021-01-15T18:47:30.346017+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:30.346017+0000","last_undegraded":"2021-01-15T18:47:30.346017+0000","last_fullsized":"2021-01-15T18:47:30.346017+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7491,"ondisk_log_size":7491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19620006,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7489,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7612","reported_seq":"7886","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.248755+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:30.248755+0000","last_peered":"2021-01-15T18:47:30.248755+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:30.248755+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7612,"ondisk_log_size":7612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19906892,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7186","reported_seq":"10059","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:28.647061+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:28.647061+0000","last_peered":"2021-01-15T18:47:28.647061+0000","last_clean":"2021-01-15T18:47:28.647061+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:28.647061+0000","last_undegraded":"2021-01-15T18:47:28.647061+0000","last_fullsized":"2021-01-15T18:47:28.647061+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7186,"ondisk_log_size":7186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19275942,"num_objects":2354,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":312699180,"num_objects":38186,"num_object_clones":0,"num_object_copies":152744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116539,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626298880,"data_stored":626298880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116566,"ondisk_log_size":116566,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373096,"kb_used_data":153092,"kb_used_omap":0,"kb_used_meta":219967,"kb_avail":93998744,"statfs":{"total":96636764160,"available":96254713856,"internally_reserved":0,"allocated":156766208,"data_stored":156513030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":225246260},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,0,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.909}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.4860000000000002},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.585}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":2.617},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":0.58699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378220,"kb_used_data":154664,"kb_used_omap":0,"kb_used_meta":223488,"kb_avail":93993620,"statfs":{"total":96636764160,"available":96249466880,"internally_reserved":0,"allocated":158375936,"data_stored":158122758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":228851712},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,7,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.383},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":6.4699999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":8.9239999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":6.2789999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.5620000000000003},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":6.6449999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373412,"kb_used_data":153544,"kb_used_omap":0,"kb_used_meta":219839,"kb_avail":93998428,"statfs":{"total":96636764160,"available":96254390272,"internally_reserved":0,"allocated":157229056,"data_stored":156975878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":225115188},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":16.263999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.7969999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":25.239999999999998},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":9.2769999999999992}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.8959999999999999},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":2.6230000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362624,"kb_used_data":153440,"kb_used_omap":0,"kb_used_meta":209151,"kb_avail":94009216,"statfs":{"total":96636764160,"available":96265437184,"internally_reserved":0,"allocated":157122560,"data_stored":156869382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":214170676},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.1660000000000004},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":5.056}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":9.6649999999999991},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":4.6710000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":9.4309999999999992},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":9.5350000000000001}]}]}]}} 2021-01-15T18:47:34.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:36.031 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:36.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2021-01-15T18:47:34.595508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":311118124,"num_objects":37993,"num_object_clones":0,"num_object_copies":151972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116732,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116759,"ondisk_log_size":116759,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1489036,"kb_used_data":610364,"kb_used_omap":2,"kb_used_meta":878525,"kb_avail":375998324,"statfs":{"total":386547056640,"available":385022283776,"internally_reserved":0,"allocated":625012736,"data_stored":624000024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":899609756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,3,12,15,8,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":125,"apply_latency_ms":125,"commit_latency_ns":125000000,"apply_latency_ns":125000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19824640,"num_objects":-2420,"num_object_clones":0,"num_object_copies":-9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004683"},"pg_stats":[{"pgid":"1.f","version":"48'7101","reported_seq":"7366","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.243879+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:30.243879+0000","last_peered":"2021-01-15T18:47:30.243879+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:30.243879+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7101,"ondisk_log_size":7101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19718144,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1157}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7158","reported_seq":"10090","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.256693+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:30.256693+0000","last_peered":"2021-01-15T18:47:30.256693+0000","last_clean":"2021-01-15T18:47:30.256693+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:30.256693+0000","last_undegraded":"2021-01-15T18:47:30.256693+0000","last_fullsized":"2021-01-15T18:47:30.256693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7158,"ondisk_log_size":7158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18866342,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7156,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7248","reported_seq":"10187","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:33.652552+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:33.652552+0000","last_peered":"2021-01-15T18:47:33.652552+0000","last_clean":"2021-01-15T18:47:33.652552+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:33.652552+0000","last_undegraded":"2021-01-15T18:47:33.652552+0000","last_fullsized":"2021-01-15T18:47:33.652552+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7248,"ondisk_log_size":7248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18378859,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7247,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7392","reported_seq":"7646","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.148951+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:30.148951+0000","last_peered":"2021-01-15T18:47:30.148951+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:30.148951+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7392,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19660800,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7431","reported_seq":"8031","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.239106+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:30.239106+0000","last_peered":"2021-01-15T18:47:30.239106+0000","last_clean":"2021-01-15T18:47:30.239106+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:30.239106+0000","last_undegraded":"2021-01-15T18:47:30.239106+0000","last_fullsized":"2021-01-15T18:47:30.239106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7431,"ondisk_log_size":7431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19636390,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7429,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7179","reported_seq":"7398","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.343168+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:30.343168+0000","last_peered":"2021-01-15T18:47:30.343168+0000","last_clean":"2021-01-15T18:47:30.343168+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:30.343168+0000","last_undegraded":"2021-01-15T18:47:30.343168+0000","last_fullsized":"2021-01-15T18:47:30.343168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7179,"ondisk_log_size":7179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19226790,"num_objects":2348,"num_object_clones":0,"num_object_copies":9392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7272","reported_seq":"8993","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.364423+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:30.364423+0000","last_peered":"2021-01-15T18:47:30.364423+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:30.364423+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7272,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20095142,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7270,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7462","reported_seq":"7626","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:33.746295+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:33.746295+0000","last_peered":"2021-01-15T18:47:33.746295+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:33.746295+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7462,"ondisk_log_size":7462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615827,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7357","reported_seq":"10300","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.161779+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:30.161779+0000","last_peered":"2021-01-15T18:47:30.161779+0000","last_clean":"2021-01-15T18:47:30.161779+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:30.161779+0000","last_undegraded":"2021-01-15T18:47:30.161779+0000","last_fullsized":"2021-01-15T18:47:30.161779+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7357,"ondisk_log_size":7357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19427411,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7377","reported_seq":"10326","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.280882+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:30.280882+0000","last_peered":"2021-01-15T18:47:30.280882+0000","last_clean":"2021-01-15T18:47:30.280882+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:30.280882+0000","last_undegraded":"2021-01-15T18:47:30.280882+0000","last_fullsized":"2021-01-15T18:47:30.280882+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7377,"ondisk_log_size":7377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19501305,"num_objects":2382,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7145","reported_seq":"7393","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.275811+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:30.275811+0000","last_peered":"2021-01-15T18:47:30.275811+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:30.275811+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7145,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19734694,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1118}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7153","reported_seq":"9990","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.350646+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:30.350646+0000","last_peered":"2021-01-15T18:47:30.350646+0000","last_clean":"2021-01-15T18:47:30.350646+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:30.350646+0000","last_undegraded":"2021-01-15T18:47:30.350646+0000","last_fullsized":"2021-01-15T18:47:30.350646+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7153,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19775654,"num_objects":2415,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7130","reported_seq":"10032","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:29.506102+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:29.506102+0000","last_peered":"2021-01-15T18:47:29.506102+0000","last_clean":"2021-01-15T18:47:29.506102+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:29.506102+0000","last_undegraded":"2021-01-15T18:47:29.506102+0000","last_fullsized":"2021-01-15T18:47:29.506102+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7130,"ondisk_log_size":7130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19210406,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7128,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7491","reported_seq":"7724","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:30.346017+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:30.346017+0000","last_peered":"2021-01-15T18:47:30.346017+0000","last_clean":"2021-01-15T18:47:30.346017+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:30.346017+0000","last_undegraded":"2021-01-15T18:47:30.346017+0000","last_fullsized":"2021-01-15T18:47:30.346017+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7491,"ondisk_log_size":7491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19620006,"num_objects":2396,"num_object_clones":0,"num_object_copies":9584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7489,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7612","reported_seq":"7886","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:30.248755+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:30.248755+0000","last_peered":"2021-01-15T18:47:30.248755+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:30.248755+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7612,"ondisk_log_size":7612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19906892,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7251","reported_seq":"10124","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:33.744097+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:33.744097+0000","last_peered":"2021-01-15T18:47:33.744097+0000","last_clean":"2021-01-15T18:47:33.744097+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:33.744097+0000","last_undegraded":"2021-01-15T18:47:33.744097+0000","last_fullsized":"2021-01-15T18:47:33.744097+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7251,"ondisk_log_size":7251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18743462,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7249,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":311118124,"num_objects":37993,"num_object_clones":0,"num_object_copies":151972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116732,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":621867008,"data_stored":621867008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116759,"ondisk_log_size":116759,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215200,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373096,"kb_used_data":153092,"kb_used_omap":0,"kb_used_meta":219967,"kb_avail":93998744,"statfs":{"total":96636764160,"available":96254713856,"internally_reserved":0,"allocated":156766208,"data_stored":156513030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":225246260},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,0,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.909}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.4860000000000002},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":2.585}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":2.617},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":0.58699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379904,"kb_used_data":150288,"kb_used_omap":0,"kb_used_meta":229568,"kb_avail":93991936,"statfs":{"total":96636764160,"available":96247742464,"internally_reserved":0,"allocated":153894912,"data_stored":153641734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":235077632},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,7,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.383},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":6.4699999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":8.9239999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":6.2789999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.5620000000000003},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":6.6449999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373412,"kb_used_data":153544,"kb_used_omap":0,"kb_used_meta":219839,"kb_avail":93998428,"statfs":{"total":96636764160,"available":96254390272,"internally_reserved":0,"allocated":157229056,"data_stored":156975878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":225115188},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,0,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":16.263999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":1.7969999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":25.239999999999998},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":9.2769999999999992}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.8959999999999999},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":2.6230000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362624,"kb_used_data":153440,"kb_used_omap":0,"kb_used_meta":209151,"kb_avail":94009216,"statfs":{"total":96636764160,"available":96265437184,"internally_reserved":0,"allocated":157122560,"data_stored":156869382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":214170676},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":5.1660000000000004},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":5.056}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":9.6649999999999991},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":4.6710000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":9.4309999999999992},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":9.5350000000000001}]}]}]}} 2021-01-15T18:47:36.081 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:36.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:36.943 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7196/148380 objects degraded (4.850%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":507,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37095,"data_bytes":303761708,"bytes_used":1542713344,"bytes_avail":385004343296,"bytes_total":386547056640,"degraded_objects":7196,"degraded_total":148380,"degraded_ratio":0.048497102035314738,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:36.956 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:36.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:37.490 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:37.524 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:37.558 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2021-01-15T18:47:36.595963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":303761708,"num_objects":37095,"num_object_clones":0,"num_object_copies":148380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37095,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117630,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14211,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117657,"ondisk_log_size":117657,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1506556,"kb_used_data":597240,"kb_used_omap":2,"kb_used_meta":909117,"kb_avail":375980804,"statfs":{"total":386547056640,"available":385004343296,"internally_reserved":0,"allocated":611573760,"data_stored":610561048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":930935964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,2,11,14,20,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":122,"apply_latency_ms":122,"commit_latency_ns":122000000,"apply_latency_ns":122000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25100288,"num_objects":-3064,"num_object_clones":0,"num_object_copies":-12256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3064,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004729"},"pg_stats":[{"pgid":"1.f","version":"48'7176","reported_seq":"7442","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:35.283522+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:35.283522+0000","last_peered":"2021-01-15T18:47:35.283522+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:35.283522+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7176,"ondisk_log_size":7176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7217","reported_seq":"10149","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.321171+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:35.321171+0000","last_peered":"2021-01-15T18:47:35.321171+0000","last_clean":"2021-01-15T18:47:35.321171+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:35.321171+0000","last_undegraded":"2021-01-15T18:47:35.321171+0000","last_fullsized":"2021-01-15T18:47:35.321171+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7217,"ondisk_log_size":7217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18383014,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7215,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7248","reported_seq":"10187","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:33.652552+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:33.652552+0000","last_peered":"2021-01-15T18:47:33.652552+0000","last_clean":"2021-01-15T18:47:33.652552+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:33.652552+0000","last_undegraded":"2021-01-15T18:47:33.652552+0000","last_fullsized":"2021-01-15T18:47:33.652552+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7248,"ondisk_log_size":7248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18378859,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7247,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7460","reported_seq":"7713","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:34.893352+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:34.893352+0000","last_peered":"2021-01-15T18:47:34.893352+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:34.893352+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7460,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":998}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7494","reported_seq":"8094","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.075429+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:35.075429+0000","last_peered":"2021-01-15T18:47:35.075429+0000","last_clean":"2021-01-15T18:47:35.075429+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:35.075429+0000","last_undegraded":"2021-01-15T18:47:35.075429+0000","last_fullsized":"2021-01-15T18:47:35.075429+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7494,"ondisk_log_size":7494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19120294,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7256","reported_seq":"7475","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.282859+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:35.282859+0000","last_peered":"2021-01-15T18:47:35.282859+0000","last_clean":"2021-01-15T18:47:35.282859+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:35.282859+0000","last_undegraded":"2021-01-15T18:47:35.282859+0000","last_fullsized":"2021-01-15T18:47:35.282859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7256,"ondisk_log_size":7256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18596006,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7254,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7354","reported_seq":"9177","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:35.275241+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:35.275241+0000","last_peered":"2021-01-15T18:47:35.275241+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:35.275241+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7354,"ondisk_log_size":7354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19423398,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":880,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7462","reported_seq":"7626","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:33.746295+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:33.746295+0000","last_peered":"2021-01-15T18:47:33.746295+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:33.746295+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7462,"ondisk_log_size":7462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615827,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7429","reported_seq":"10372","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.145622+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:35.145622+0000","last_peered":"2021-01-15T18:47:35.145622+0000","last_clean":"2021-01-15T18:47:35.145622+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:35.145622+0000","last_undegraded":"2021-01-15T18:47:35.145622+0000","last_fullsized":"2021-01-15T18:47:35.145622+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7429,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18837587,"num_objects":2300,"num_object_clones":0,"num_object_copies":9200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7440","reported_seq":"10386","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.273129+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:35.273129+0000","last_peered":"2021-01-15T18:47:35.273129+0000","last_clean":"2021-01-15T18:47:35.273129+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:35.273129+0000","last_undegraded":"2021-01-15T18:47:35.273129+0000","last_fullsized":"2021-01-15T18:47:35.273129+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7440,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18985209,"num_objects":2319,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7437,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7211","reported_seq":"7458","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:35.268787+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:35.268787+0000","last_peered":"2021-01-15T18:47:35.268787+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:35.268787+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7211,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19194022,"num_objects":2344,"num_object_clones":0,"num_object_copies":9376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7219","reported_seq":"10056","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.213120+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:35.213120+0000","last_peered":"2021-01-15T18:47:35.213120+0000","last_clean":"2021-01-15T18:47:35.213120+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:35.213120+0000","last_undegraded":"2021-01-15T18:47:35.213120+0000","last_fullsized":"2021-01-15T18:47:35.213120+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7219,"ondisk_log_size":7219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19234982,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7185","reported_seq":"10088","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:34.447900+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:34.447900+0000","last_peered":"2021-01-15T18:47:34.447900+0000","last_clean":"2021-01-15T18:47:34.447900+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:34.447900+0000","last_undegraded":"2021-01-15T18:47:34.447900+0000","last_fullsized":"2021-01-15T18:47:34.447900+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7185,"ondisk_log_size":7185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18759846,"num_objects":2291,"num_object_clones":0,"num_object_copies":9164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7571","reported_seq":"7805","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:35.341102+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:35.341102+0000","last_peered":"2021-01-15T18:47:35.341102+0000","last_clean":"2021-01-15T18:47:35.341102+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:35.341102+0000","last_undegraded":"2021-01-15T18:47:35.341102+0000","last_fullsized":"2021-01-15T18:47:35.341102+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7571,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18964646,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7684","reported_seq":"7958","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:35.272913+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:35.272913+0000","last_peered":"2021-01-15T18:47:35.272913+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:35.272913+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7684,"ondisk_log_size":7684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19317068,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7681,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7251","reported_seq":"10124","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:33.744097+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:33.744097+0000","last_peered":"2021-01-15T18:47:33.744097+0000","last_clean":"2021-01-15T18:47:33.744097+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:33.744097+0000","last_undegraded":"2021-01-15T18:47:33.744097+0000","last_fullsized":"2021-01-15T18:47:33.744097+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7251,"ondisk_log_size":7251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18743462,"num_objects":2289,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7249,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":303761708,"num_objects":37095,"num_object_clones":0,"num_object_copies":148380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37095,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117630,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14211,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":608518144,"data_stored":608518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117657,"ondisk_log_size":117657,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215201,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378908,"kb_used_data":148728,"kb_used_omap":0,"kb_used_meta":230143,"kb_avail":93992932,"statfs":{"total":96636764160,"available":96248762368,"internally_reserved":0,"allocated":152297472,"data_stored":152044294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":235666484},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.909}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.1819999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":1.2749999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379904,"kb_used_data":150288,"kb_used_omap":0,"kb_used_meta":229568,"kb_avail":93991936,"statfs":{"total":96636764160,"available":96247742464,"internally_reserved":0,"allocated":153894912,"data_stored":153641734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":235077632},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,7,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.383},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":6.4699999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":2.8239999999999998,"15min":2.4649999999999999},"min":{"1min":0.45000000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":13.586,"5min":20.885000000000002,"15min":20.885000000000002},"last":8.9239999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.746,"15min":1.9610000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":3.6669999999999998,"5min":20.98,"15min":20.98},"last":6.2789999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.5620000000000003},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":6.6449999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379232,"kb_used_data":149188,"kb_used_omap":0,"kb_used_meta":230015,"kb_avail":93992608,"statfs":{"total":96636764160,"available":96248430592,"internally_reserved":0,"allocated":152768512,"data_stored":152515334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":235535412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:46:35 2021","interfaces":[{"interface":"back","average":{"1min":4.4489999999999998,"5min":3.548,"15min":3.1419999999999999},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":32.317,"5min":35.406999999999996,"15min":35.406999999999996},"last":18.088999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":4.4029999999999996,"15min":2.863},"min":{"1min":0.46600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":15.804,"5min":44.072000000000003,"15min":44.072000000000003},"last":8.6240000000000006}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":2.0409999999999999},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":13.262},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":17.001999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368512,"kb_used_data":149036,"kb_used_omap":0,"kb_used_meta":219391,"kb_avail":94003328,"statfs":{"total":96636764160,"available":96259407872,"internally_reserved":0,"allocated":152612864,"data_stored":152359686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":224656436},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":2.4740000000000002},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.745}]}]}]}} 2021-01-15T18:47:40.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:41.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:41.865 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2021-01-15T18:47:40.598914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":295668012,"num_objects":36107,"num_object_clones":0,"num_object_copies":144428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118618,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14251,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118645,"ondisk_log_size":118645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1506124,"kb_used_data":581368,"kb_used_omap":2,"kb_used_meta":924541,"kb_avail":375981236,"statfs":{"total":386547056640,"available":385004785664,"internally_reserved":0,"allocated":595320832,"data_stored":594308120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":946730140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,9,10,24,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26476544,"num_objects":-3232,"num_object_clones":0,"num_object_copies":-12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006759"},"pg_stats":[{"pgid":"1.f","version":"48'7244","reported_seq":"7509","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.342487+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:40.342487+0000","last_peered":"2021-01-15T18:47:40.342487+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:40.342487+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7244,"ondisk_log_size":7244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7277","reported_seq":"10209","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.110675+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:40.110675+0000","last_peered":"2021-01-15T18:47:40.110675+0000","last_clean":"2021-01-15T18:47:40.110675+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:40.110675+0000","last_undegraded":"2021-01-15T18:47:40.110675+0000","last_fullsized":"2021-01-15T18:47:40.110675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7277,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17891494,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7301","reported_seq":"10240","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:38.793892+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:38.793892+0000","last_peered":"2021-01-15T18:47:38.793892+0000","last_clean":"2021-01-15T18:47:38.793892+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:38.793892+0000","last_undegraded":"2021-01-15T18:47:38.793892+0000","last_fullsized":"2021-01-15T18:47:38.793892+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7301,"ondisk_log_size":7301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17944683,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7300,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7521","reported_seq":"7775","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.307753+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:40.307753+0000","last_peered":"2021-01-15T18:47:40.307753+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:40.307753+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7521,"ondisk_log_size":7521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18604032,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7551","reported_seq":"8151","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.349084+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:40.349084+0000","last_peered":"2021-01-15T18:47:40.349084+0000","last_clean":"2021-01-15T18:47:40.349084+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:40.349084+0000","last_undegraded":"2021-01-15T18:47:40.349084+0000","last_fullsized":"2021-01-15T18:47:40.349084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7551,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18653350,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7328","reported_seq":"7547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.335130+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:40.335130+0000","last_peered":"2021-01-15T18:47:40.335130+0000","last_clean":"2021-01-15T18:47:40.335130+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:40.335130+0000","last_undegraded":"2021-01-15T18:47:40.335130+0000","last_fullsized":"2021-01-15T18:47:40.335130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7328,"ondisk_log_size":7328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18006182,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7418","reported_seq":"9321","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.411340+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:40.411340+0000","last_peered":"2021-01-15T18:47:40.411340+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:40.411340+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7418,"ondisk_log_size":7418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18899110,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7515","reported_seq":"7679","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:38.602712+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:38.602712+0000","last_peered":"2021-01-15T18:47:38.602712+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:38.602712+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7515,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19181651,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":929}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7498","reported_seq":"10441","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.364822+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:40.364822+0000","last_peered":"2021-01-15T18:47:40.364822+0000","last_clean":"2021-01-15T18:47:40.364822+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:40.364822+0000","last_undegraded":"2021-01-15T18:47:40.364822+0000","last_fullsized":"2021-01-15T18:47:40.364822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7498,"ondisk_log_size":7498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18272339,"num_objects":2231,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7508","reported_seq":"10456","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.297369+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:40.297369+0000","last_peered":"2021-01-15T18:47:40.297369+0000","last_clean":"2021-01-15T18:47:40.297369+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:40.297369+0000","last_undegraded":"2021-01-15T18:47:40.297369+0000","last_fullsized":"2021-01-15T18:47:40.297369+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7508,"ondisk_log_size":7508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18428153,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7277","reported_seq":"7525","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.391045+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:40.391045+0000","last_peered":"2021-01-15T18:47:40.391045+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:40.391045+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7277,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18653350,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7284","reported_seq":"10121","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.297138+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:40.297138+0000","last_peered":"2021-01-15T18:47:40.297138+0000","last_clean":"2021-01-15T18:47:40.297138+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:40.297138+0000","last_undegraded":"2021-01-15T18:47:40.297138+0000","last_fullsized":"2021-01-15T18:47:40.297138+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7284,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18702502,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7282,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7240","reported_seq":"10143","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:39.337410+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:39.337410+0000","last_peered":"2021-01-15T18:47:39.337410+0000","last_clean":"2021-01-15T18:47:39.337410+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:39.337410+0000","last_undegraded":"2021-01-15T18:47:39.337410+0000","last_fullsized":"2021-01-15T18:47:39.337410+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7240,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18309286,"num_objects":2236,"num_object_clones":0,"num_object_copies":8944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7623","reported_seq":"7857","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.333680+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:40.333680+0000","last_peered":"2021-01-15T18:47:40.333680+0000","last_clean":"2021-01-15T18:47:40.333680+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:40.333680+0000","last_undegraded":"2021-01-15T18:47:40.333680+0000","last_fullsized":"2021-01-15T18:47:40.333680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7623,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18538662,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7749","reported_seq":"8021","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.173769+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:40.173769+0000","last_peered":"2021-01-15T18:47:40.173769+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:40.173769+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7749,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18784588,"num_objects":2295,"num_object_clones":0,"num_object_copies":9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7746,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7311","reported_seq":"10184","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:38.790499+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:38.790499+0000","last_peered":"2021-01-15T18:47:38.790499+0000","last_clean":"2021-01-15T18:47:38.790499+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:38.790499+0000","last_undegraded":"2021-01-15T18:47:38.790499+0000","last_fullsized":"2021-01-15T18:47:38.790499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7311,"ondisk_log_size":7311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18251942,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":295668012,"num_objects":36107,"num_object_clones":0,"num_object_copies":144428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118618,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14251,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":592195584,"data_stored":592195584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118645,"ondisk_log_size":118645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380848,"kb_used_data":144772,"kb_used_omap":0,"kb_used_meta":236031,"kb_avail":93990992,"statfs":{"total":96636764160,"available":96246775808,"internally_reserved":0,"allocated":148246528,"data_stored":147993350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":241695796},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,5,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369616,"kb_used_data":146368,"kb_used_omap":0,"kb_used_meta":223168,"kb_avail":94002224,"statfs":{"total":96636764160,"available":96258277376,"internally_reserved":0,"allocated":149880832,"data_stored":149627654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":228524032},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.2310000000000001},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":1.325}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.874},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":0.97699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385232,"kb_used_data":145184,"kb_used_omap":0,"kb_used_meta":239999,"kb_avail":93986608,"statfs":{"total":96636764160,"available":96242286592,"internally_reserved":0,"allocated":148668416,"data_stored":148415238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245759028},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.59099999999999997},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":3.8660000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":3.383},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":3.4039999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":3.4790000000000001},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":3.5659999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370428,"kb_used_data":145044,"kb_used_omap":0,"kb_used_meta":225343,"kb_avail":94001412,"statfs":{"total":96636764160,"available":96257445888,"internally_reserved":0,"allocated":148525056,"data_stored":148271878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":230751284},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.5460000000000003},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":4.665},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":7.1929999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.80000000000000004}]}]}]}} 2021-01-15T18:47:41.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:43.111 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:43.111 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7156/144428 objects degraded (4.955%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36107,"data_bytes":295668012,"bytes_used":1542270976,"bytes_avail":385004785664,"bytes_total":386547056640,"degraded_objects":7156,"degraded_total":144428,"degraded_ratio":0.049547179217326279,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:43.164 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:43.164 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:43.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:43.798 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:43.830 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2021-01-15T18:47:42.599399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":295668012,"num_objects":36107,"num_object_clones":0,"num_object_copies":144428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118618,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14251,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118645,"ondisk_log_size":118645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1506124,"kb_used_data":581368,"kb_used_omap":2,"kb_used_meta":924541,"kb_avail":375981236,"statfs":{"total":386547056640,"available":385004785664,"internally_reserved":0,"allocated":595320832,"data_stored":594308120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":946730140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,9,10,24,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17031168,"num_objects":-2079,"num_object_clones":0,"num_object_copies":-8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006800"},"pg_stats":[{"pgid":"1.f","version":"48'7244","reported_seq":"7509","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.342487+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:40.342487+0000","last_peered":"2021-01-15T18:47:40.342487+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:40.342487+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7244,"ondisk_log_size":7244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18546688,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1014}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7277","reported_seq":"10209","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.110675+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:40.110675+0000","last_peered":"2021-01-15T18:47:40.110675+0000","last_clean":"2021-01-15T18:47:40.110675+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:40.110675+0000","last_undegraded":"2021-01-15T18:47:40.110675+0000","last_fullsized":"2021-01-15T18:47:40.110675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7277,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17891494,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7301","reported_seq":"10240","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:38.793892+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:38.793892+0000","last_peered":"2021-01-15T18:47:38.793892+0000","last_clean":"2021-01-15T18:47:38.793892+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:38.793892+0000","last_undegraded":"2021-01-15T18:47:38.793892+0000","last_fullsized":"2021-01-15T18:47:38.793892+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7301,"ondisk_log_size":7301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17944683,"num_objects":2192,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7300,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7521","reported_seq":"7775","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.307753+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:40.307753+0000","last_peered":"2021-01-15T18:47:40.307753+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:40.307753+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7521,"ondisk_log_size":7521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18604032,"num_objects":2271,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7521,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7551","reported_seq":"8151","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.349084+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:40.349084+0000","last_peered":"2021-01-15T18:47:40.349084+0000","last_clean":"2021-01-15T18:47:40.349084+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:40.349084+0000","last_undegraded":"2021-01-15T18:47:40.349084+0000","last_fullsized":"2021-01-15T18:47:40.349084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7551,"ondisk_log_size":7551,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18653350,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7328","reported_seq":"7547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.335130+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:40.335130+0000","last_peered":"2021-01-15T18:47:40.335130+0000","last_clean":"2021-01-15T18:47:40.335130+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:40.335130+0000","last_undegraded":"2021-01-15T18:47:40.335130+0000","last_fullsized":"2021-01-15T18:47:40.335130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7328,"ondisk_log_size":7328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18006182,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7418","reported_seq":"9321","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.411340+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:40.411340+0000","last_peered":"2021-01-15T18:47:40.411340+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:40.411340+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7418,"ondisk_log_size":7418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18899110,"num_objects":2308,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7515","reported_seq":"7679","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:38.602712+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:38.602712+0000","last_peered":"2021-01-15T18:47:38.602712+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:38.602712+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7515,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19181651,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7514,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":929}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7498","reported_seq":"10441","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.364822+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:40.364822+0000","last_peered":"2021-01-15T18:47:40.364822+0000","last_clean":"2021-01-15T18:47:40.364822+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:40.364822+0000","last_undegraded":"2021-01-15T18:47:40.364822+0000","last_fullsized":"2021-01-15T18:47:40.364822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7498,"ondisk_log_size":7498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18272339,"num_objects":2231,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7508","reported_seq":"10456","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.297369+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:40.297369+0000","last_peered":"2021-01-15T18:47:40.297369+0000","last_clean":"2021-01-15T18:47:40.297369+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:40.297369+0000","last_undegraded":"2021-01-15T18:47:40.297369+0000","last_fullsized":"2021-01-15T18:47:40.297369+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7508,"ondisk_log_size":7508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18428153,"num_objects":2251,"num_object_clones":0,"num_object_copies":9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7277","reported_seq":"7525","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.391045+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:40.391045+0000","last_peered":"2021-01-15T18:47:40.391045+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:40.391045+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7277,"ondisk_log_size":7277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18653350,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":986}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7284","reported_seq":"10121","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.297138+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:40.297138+0000","last_peered":"2021-01-15T18:47:40.297138+0000","last_clean":"2021-01-15T18:47:40.297138+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:40.297138+0000","last_undegraded":"2021-01-15T18:47:40.297138+0000","last_fullsized":"2021-01-15T18:47:40.297138+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7284,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18702502,"num_objects":2284,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7282,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7240","reported_seq":"10143","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:39.337410+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:39.337410+0000","last_peered":"2021-01-15T18:47:39.337410+0000","last_clean":"2021-01-15T18:47:39.337410+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:39.337410+0000","last_undegraded":"2021-01-15T18:47:39.337410+0000","last_fullsized":"2021-01-15T18:47:39.337410+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7240,"ondisk_log_size":7240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18309286,"num_objects":2236,"num_object_clones":0,"num_object_copies":8944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7623","reported_seq":"7857","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:40.333680+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:40.333680+0000","last_peered":"2021-01-15T18:47:40.333680+0000","last_clean":"2021-01-15T18:47:40.333680+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:40.333680+0000","last_undegraded":"2021-01-15T18:47:40.333680+0000","last_fullsized":"2021-01-15T18:47:40.333680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7623,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18538662,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7749","reported_seq":"8021","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:40.173769+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:40.173769+0000","last_peered":"2021-01-15T18:47:40.173769+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:40.173769+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7749,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18784588,"num_objects":2295,"num_object_clones":0,"num_object_copies":9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7746,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7311","reported_seq":"10184","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:38.790499+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:38.790499+0000","last_peered":"2021-01-15T18:47:38.790499+0000","last_clean":"2021-01-15T18:47:38.790499+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:38.790499+0000","last_undegraded":"2021-01-15T18:47:38.790499+0000","last_fullsized":"2021-01-15T18:47:38.790499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7311,"ondisk_log_size":7311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18251942,"num_objects":2229,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":295668012,"num_objects":36107,"num_object_clones":0,"num_object_copies":144428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118618,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14251,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":592195584,"data_stored":592195584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118645,"ondisk_log_size":118645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215202,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380848,"kb_used_data":144772,"kb_used_omap":0,"kb_used_meta":236031,"kb_avail":93990992,"statfs":{"total":96636764160,"available":96246775808,"internally_reserved":0,"allocated":148246528,"data_stored":147993350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":241695796},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,5,1,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:46:39 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":4.2149999999999999,"15min":3.4529999999999998},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.585,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.647,"5min":4.0750000000000002,"15min":2.6509999999999998},"min":{"1min":0.68899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.7510000000000003,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369616,"kb_used_data":146368,"kb_used_omap":0,"kb_used_meta":223168,"kb_avail":94002224,"statfs":{"total":96636764160,"available":96258277376,"internally_reserved":0,"allocated":149880832,"data_stored":149627654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":228524032},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.2310000000000001},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":1.325}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.874},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":0.97699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385232,"kb_used_data":145184,"kb_used_omap":0,"kb_used_meta":239999,"kb_avail":93986608,"statfs":{"total":96636764160,"available":96242286592,"internally_reserved":0,"allocated":148668416,"data_stored":148415238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245759028},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.59099999999999997},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":3.8660000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":3.383},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":3.4039999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":3.4790000000000001},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":3.5659999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370428,"kb_used_data":145044,"kb_used_omap":0,"kb_used_meta":225343,"kb_avail":94001412,"statfs":{"total":96636764160,"available":96257445888,"internally_reserved":0,"allocated":148525056,"data_stored":148271878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":230751284},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:42 2021","interfaces":[{"interface":"back","average":{"1min":3.899,"5min":4.6239999999999997,"15min":3.339},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":14.013,"5min":35.850999999999999,"15min":35.850999999999999},"last":7.5460000000000003},{"interface":"front","average":{"1min":1.175,"5min":2.3690000000000002,"15min":2.024},"min":{"1min":0.45600000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.0039999999999996,"5min":32.216999999999999,"15min":32.216999999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.5129999999999999,"15min":2.6539999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.8260000000000005,"5min":21.791,"15min":21.791},"last":4.665},{"interface":"front","average":{"1min":1.476,"5min":2.0619999999999998,"15min":1.7889999999999999},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.8140000000000001,"5min":19.381,"15min":19.381},"last":7.1929999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:46:41 2021","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":3.0640000000000001,"15min":3.8159999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":11.231999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":2.4729999999999999,"15min":2.5790000000000002},"min":{"1min":0.67400000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":11.332000000000001,"5min":13.478999999999999,"15min":13.478999999999999},"last":0.80000000000000004}]}]}]}} 2021-01-15T18:47:46.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:47.644 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:47.657 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:47.683 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2021-01-15T18:47:46.602815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286935340,"num_objects":35041,"num_object_clones":0,"num_object_copies":140164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119684,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14302,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119711,"ondisk_log_size":119711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1476892,"kb_used_data":565240,"kb_used_omap":2,"kb_used_meta":911485,"kb_avail":376010468,"statfs":{"total":386547056640,"available":385034719232,"internally_reserved":0,"allocated":578805760,"data_stored":577793048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":933360796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,10,10,15,19,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24182784,"num_objects":-2952,"num_object_clones":0,"num_object_copies":-11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2952,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007306"},"pg_stats":[{"pgid":"1.f","version":"48'7309","reported_seq":"7575","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:44.903919+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:44.903919+0000","last_peered":"2021-01-15T18:47:44.903919+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:44.903919+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7309,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18014208,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7352","reported_seq":"10283","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.230858+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:45.230858+0000","last_peered":"2021-01-15T18:47:45.230858+0000","last_clean":"2021-01-15T18:47:45.230858+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:45.230858+0000","last_undegraded":"2021-01-15T18:47:45.230858+0000","last_fullsized":"2021-01-15T18:47:45.230858+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7352,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17277094,"num_objects":2110,"num_object_clones":0,"num_object_copies":8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7364","reported_seq":"10303","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:43.758516+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:43.758516+0000","last_peered":"2021-01-15T18:47:43.758516+0000","last_clean":"2021-01-15T18:47:43.758516+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:43.758516+0000","last_undegraded":"2021-01-15T18:47:43.758516+0000","last_fullsized":"2021-01-15T18:47:43.758516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7364,"ondisk_log_size":7364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17428587,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7363,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7588","reported_seq":"7842","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.362604+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:45.362604+0000","last_peered":"2021-01-15T18:47:45.362604+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:45.362604+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7588,"ondisk_log_size":7588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18055168,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7623","reported_seq":"8223","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.442091+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:45.442091+0000","last_peered":"2021-01-15T18:47:45.442091+0000","last_clean":"2021-01-15T18:47:45.442091+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:45.442091+0000","last_undegraded":"2021-01-15T18:47:45.442091+0000","last_fullsized":"2021-01-15T18:47:45.442091+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7623,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18063526,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7399","reported_seq":"7618","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.395929+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:45.395929+0000","last_peered":"2021-01-15T18:47:45.395929+0000","last_clean":"2021-01-15T18:47:45.395929+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:45.395929+0000","last_undegraded":"2021-01-15T18:47:45.395929+0000","last_fullsized":"2021-01-15T18:47:45.395929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7399,"ondisk_log_size":7399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17424550,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7397,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7474","reported_seq":"9479","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.379292+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:45.379292+0000","last_peered":"2021-01-15T18:47:45.379292+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:45.379292+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7474,"ondisk_log_size":7474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18440358,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1827}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7590","reported_seq":"7754","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:43.759984+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:43.759984+0000","last_peered":"2021-01-15T18:47:43.759984+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:43.759984+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7590,"ondisk_log_size":7590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18567251,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7589,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7564","reported_seq":"10505","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.416259+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:45.416259+0000","last_peered":"2021-01-15T18:47:45.416259+0000","last_clean":"2021-01-15T18:47:45.416259+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:45.416259+0000","last_undegraded":"2021-01-15T18:47:45.416259+0000","last_fullsized":"2021-01-15T18:47:45.416259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7564,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17731667,"num_objects":2165,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7584","reported_seq":"10533","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.378524+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:45.378524+0000","last_peered":"2021-01-15T18:47:45.378524+0000","last_clean":"2021-01-15T18:47:45.378524+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:45.378524+0000","last_undegraded":"2021-01-15T18:47:45.378524+0000","last_fullsized":"2021-01-15T18:47:45.378524+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7584,"ondisk_log_size":7584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17805561,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7581,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7353","reported_seq":"7599","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.378801+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:45.378801+0000","last_peered":"2021-01-15T18:47:45.378801+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:45.378801+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7353,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18030758,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7343","reported_seq":"10180","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.165977+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:45.165977+0000","last_peered":"2021-01-15T18:47:45.165977+0000","last_clean":"2021-01-15T18:47:45.165977+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:45.165977+0000","last_undegraded":"2021-01-15T18:47:45.165977+0000","last_fullsized":"2021-01-15T18:47:45.165977+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7343,"ondisk_log_size":7343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18219174,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7341,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7301","reported_seq":"10204","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:44.475476+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:44.475476+0000","last_peered":"2021-01-15T18:47:44.475476+0000","last_clean":"2021-01-15T18:47:44.475476+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:44.475476+0000","last_undegraded":"2021-01-15T18:47:44.475476+0000","last_fullsized":"2021-01-15T18:47:44.475476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7301,"ondisk_log_size":7301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17809574,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7687","reported_seq":"7921","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.380351+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:45.380351+0000","last_peered":"2021-01-15T18:47:45.380351+0000","last_clean":"2021-01-15T18:47:45.380351+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:45.380351+0000","last_undegraded":"2021-01-15T18:47:45.380351+0000","last_fullsized":"2021-01-15T18:47:45.380351+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7687,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18014374,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7808","reported_seq":"8080","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.459456+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:45.459456+0000","last_peered":"2021-01-15T18:47:45.459456+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:45.459456+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7808,"ondisk_log_size":7808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18301260,"num_objects":2236,"num_object_clones":0,"num_object_copies":8944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7372","reported_seq":"10244","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:43.850416+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:43.850416+0000","last_peered":"2021-01-15T18:47:43.850416+0000","last_clean":"2021-01-15T18:47:43.850416+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:43.850416+0000","last_undegraded":"2021-01-15T18:47:43.850416+0000","last_fullsized":"2021-01-15T18:47:43.850416+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7372,"ondisk_log_size":7372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17752230,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286935340,"num_objects":35041,"num_object_clones":0,"num_object_copies":140164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119684,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14302,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575078400,"data_stored":575078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119711,"ondisk_log_size":119711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370384,"kb_used_data":140628,"kb_used_omap":0,"kb_used_meta":229695,"kb_avail":94001456,"statfs":{"total":96636764160,"available":96257490944,"internally_reserved":0,"allocated":144003072,"data_stored":143749894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":235207732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.7989999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.99199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":3.6859999999999999},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":6.4379999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375380,"kb_used_data":141948,"kb_used_omap":0,"kb_used_meta":233344,"kb_avail":93996460,"statfs":{"total":96636764160,"available":96252375040,"internally_reserved":0,"allocated":145354752,"data_stored":145101574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":238944256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.159},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.254}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":8.1460000000000008},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":12.074999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.4339999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371068,"kb_used_data":141748,"kb_used_omap":0,"kb_used_meta":229311,"kb_avail":94000772,"statfs":{"total":96636764160,"available":96256790528,"internally_reserved":0,"allocated":145149952,"data_stored":144896774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234814516},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":12.831},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":15.106}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":1.2509999999999999},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.0389999999999997},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.96399999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360060,"kb_used_data":140916,"kb_used_omap":0,"kb_used_meta":219135,"kb_avail":94011780,"statfs":{"total":96636764160,"available":96268062720,"internally_reserved":0,"allocated":144297984,"data_stored":144044806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":224394292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":3.7570000000000001},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":2.3919999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.4970000000000001},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.117},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.206}]}]}]}} 2021-01-15T18:47:47.684 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:48.851 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:48.852 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7105/140164 objects degraded (5.069%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":519,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35041,"data_bytes":286935340,"bytes_used":1512337408,"bytes_avail":385034719232,"bytes_total":386547056640,"degraded_objects":7105,"degraded_total":140164,"degraded_ratio":0.050690619559944056,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:48.941 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:48.941 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:49.990 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:49.998 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:50.068 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2021-01-15T18:47:48.603324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286935340,"num_objects":35041,"num_object_clones":0,"num_object_copies":140164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119684,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14302,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119711,"ondisk_log_size":119711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1476892,"kb_used_data":565240,"kb_used_omap":2,"kb_used_meta":911485,"kb_avail":376010468,"statfs":{"total":386547056640,"available":385034719232,"internally_reserved":0,"allocated":578805760,"data_stored":577793048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":933360796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,10,10,15,19,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007360"},"pg_stats":[{"pgid":"1.f","version":"48'7309","reported_seq":"7575","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:44.903919+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:44.903919+0000","last_peered":"2021-01-15T18:47:44.903919+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:44.903919+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7309,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18014208,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7352","reported_seq":"10283","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.230858+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:45.230858+0000","last_peered":"2021-01-15T18:47:45.230858+0000","last_clean":"2021-01-15T18:47:45.230858+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:45.230858+0000","last_undegraded":"2021-01-15T18:47:45.230858+0000","last_fullsized":"2021-01-15T18:47:45.230858+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7352,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17277094,"num_objects":2110,"num_object_clones":0,"num_object_copies":8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7364","reported_seq":"10303","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:43.758516+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:43.758516+0000","last_peered":"2021-01-15T18:47:43.758516+0000","last_clean":"2021-01-15T18:47:43.758516+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:43.758516+0000","last_undegraded":"2021-01-15T18:47:43.758516+0000","last_fullsized":"2021-01-15T18:47:43.758516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7364,"ondisk_log_size":7364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17428587,"num_objects":2129,"num_object_clones":0,"num_object_copies":8516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7363,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7588","reported_seq":"7842","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.362604+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:45.362604+0000","last_peered":"2021-01-15T18:47:45.362604+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:45.362604+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7588,"ondisk_log_size":7588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18055168,"num_objects":2204,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7623","reported_seq":"8223","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.442091+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:45.442091+0000","last_peered":"2021-01-15T18:47:45.442091+0000","last_clean":"2021-01-15T18:47:45.442091+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:45.442091+0000","last_undegraded":"2021-01-15T18:47:45.442091+0000","last_fullsized":"2021-01-15T18:47:45.442091+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7623,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18063526,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7399","reported_seq":"7618","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.395929+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:45.395929+0000","last_peered":"2021-01-15T18:47:45.395929+0000","last_clean":"2021-01-15T18:47:45.395929+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:45.395929+0000","last_undegraded":"2021-01-15T18:47:45.395929+0000","last_fullsized":"2021-01-15T18:47:45.395929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7399,"ondisk_log_size":7399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17424550,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7397,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7474","reported_seq":"9479","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.379292+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:45.379292+0000","last_peered":"2021-01-15T18:47:45.379292+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:45.379292+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7474,"ondisk_log_size":7474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18440358,"num_objects":2252,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1827}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7590","reported_seq":"7754","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:43.759984+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:43.759984+0000","last_peered":"2021-01-15T18:47:43.759984+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:43.759984+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7590,"ondisk_log_size":7590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18567251,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7589,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7564","reported_seq":"10505","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.416259+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:45.416259+0000","last_peered":"2021-01-15T18:47:45.416259+0000","last_clean":"2021-01-15T18:47:45.416259+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:45.416259+0000","last_undegraded":"2021-01-15T18:47:45.416259+0000","last_fullsized":"2021-01-15T18:47:45.416259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7564,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17731667,"num_objects":2165,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7584","reported_seq":"10533","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.378524+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:45.378524+0000","last_peered":"2021-01-15T18:47:45.378524+0000","last_clean":"2021-01-15T18:47:45.378524+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:45.378524+0000","last_undegraded":"2021-01-15T18:47:45.378524+0000","last_fullsized":"2021-01-15T18:47:45.378524+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7584,"ondisk_log_size":7584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17805561,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7581,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7353","reported_seq":"7599","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.378801+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:45.378801+0000","last_peered":"2021-01-15T18:47:45.378801+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:45.378801+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7353,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18030758,"num_objects":2202,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":910}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7343","reported_seq":"10180","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.165977+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:45.165977+0000","last_peered":"2021-01-15T18:47:45.165977+0000","last_clean":"2021-01-15T18:47:45.165977+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:45.165977+0000","last_undegraded":"2021-01-15T18:47:45.165977+0000","last_fullsized":"2021-01-15T18:47:45.165977+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7343,"ondisk_log_size":7343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18219174,"num_objects":2225,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7341,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7301","reported_seq":"10204","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:44.475476+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:44.475476+0000","last_peered":"2021-01-15T18:47:44.475476+0000","last_clean":"2021-01-15T18:47:44.475476+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:44.475476+0000","last_undegraded":"2021-01-15T18:47:44.475476+0000","last_fullsized":"2021-01-15T18:47:44.475476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7301,"ondisk_log_size":7301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17809574,"num_objects":2175,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7687","reported_seq":"7921","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:45.380351+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:45.380351+0000","last_peered":"2021-01-15T18:47:45.380351+0000","last_clean":"2021-01-15T18:47:45.380351+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:45.380351+0000","last_undegraded":"2021-01-15T18:47:45.380351+0000","last_fullsized":"2021-01-15T18:47:45.380351+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7687,"ondisk_log_size":7687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18014374,"num_objects":2200,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7685,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7808","reported_seq":"8080","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:45.459456+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:45.459456+0000","last_peered":"2021-01-15T18:47:45.459456+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:45.459456+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7808,"ondisk_log_size":7808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18301260,"num_objects":2236,"num_object_clones":0,"num_object_copies":8944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7372","reported_seq":"10244","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:43.850416+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:43.850416+0000","last_peered":"2021-01-15T18:47:43.850416+0000","last_clean":"2021-01-15T18:47:43.850416+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:43.850416+0000","last_undegraded":"2021-01-15T18:47:43.850416+0000","last_fullsized":"2021-01-15T18:47:43.850416+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7372,"ondisk_log_size":7372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17752230,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286935340,"num_objects":35041,"num_object_clones":0,"num_object_copies":140164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119684,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14302,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":575078400,"data_stored":575078400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119711,"ondisk_log_size":119711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215203,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370384,"kb_used_data":140628,"kb_used_omap":0,"kb_used_meta":229695,"kb_avail":94001456,"statfs":{"total":96636764160,"available":96257490944,"internally_reserved":0,"allocated":144003072,"data_stored":143749894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":235207732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.7989999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.99199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":3.6859999999999999},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":6.4379999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375380,"kb_used_data":141948,"kb_used_omap":0,"kb_used_meta":233344,"kb_avail":93996460,"statfs":{"total":96636764160,"available":96252375040,"internally_reserved":0,"allocated":145354752,"data_stored":145101574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":238944256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.159},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":1.254}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":8.1460000000000008},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":12.074999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":1.4339999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371068,"kb_used_data":141748,"kb_used_omap":0,"kb_used_meta":229311,"kb_avail":94000772,"statfs":{"total":96636764160,"available":96256790528,"internally_reserved":0,"allocated":145149952,"data_stored":144896774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234814516},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":12.831},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":15.106}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":1.2509999999999999},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.0389999999999997},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.96399999999999997}]}]},{"osd":1,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360060,"kb_used_data":140916,"kb_used_omap":0,"kb_used_meta":219135,"kb_avail":94011780,"statfs":{"total":96636764160,"available":96268062720,"internally_reserved":0,"allocated":144297984,"data_stored":144044806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":224394292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":3.7570000000000001},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":2.3919999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.4970000000000001},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.117},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.206}]}]}]}} 2021-01-15T18:47:53.070 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:54.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:54.353 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:54.373 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2021-01-15T18:47:52.604178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278489388,"num_objects":34010,"num_object_clones":0,"num_object_copies":136040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120715,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14358,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120742,"ondisk_log_size":120742,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1480452,"kb_used_data":548872,"kb_used_omap":2,"kb_used_meta":931389,"kb_avail":376006908,"statfs":{"total":386547056640,"available":385031073792,"internally_reserved":0,"allocated":562044928,"data_stored":561032216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":953742492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,9,11,19,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17178624,"num_objects":-2097,"num_object_clones":0,"num_object_copies":-8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005263"},"pg_stats":[{"pgid":"1.f","version":"48'7366","reported_seq":"7631","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:50.386209+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:50.386209+0000","last_peered":"2021-01-15T18:47:50.386209+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:50.386209+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7366,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17547264,"num_objects":2142,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7428","reported_seq":"10359","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.347750+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:50.347750+0000","last_peered":"2021-01-15T18:47:50.347750+0000","last_clean":"2021-01-15T18:47:50.347750+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:50.347750+0000","last_undegraded":"2021-01-15T18:47:50.347750+0000","last_fullsized":"2021-01-15T18:47:50.347750+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7428,"ondisk_log_size":7428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16654502,"num_objects":2034,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7421","reported_seq":"10360","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:48.680096+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:48.680096+0000","last_peered":"2021-01-15T18:47:48.680096+0000","last_clean":"2021-01-15T18:47:48.680096+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:48.680096+0000","last_undegraded":"2021-01-15T18:47:48.680096+0000","last_fullsized":"2021-01-15T18:47:48.680096+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7421,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16961643,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7420,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7640","reported_seq":"7894","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:50.338647+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:50.338647+0000","last_peered":"2021-01-15T18:47:50.338647+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:50.338647+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7640,"ondisk_log_size":7640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17629184,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7640,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7697","reported_seq":"8296","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.230770+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:50.230770+0000","last_peered":"2021-01-15T18:47:50.230770+0000","last_clean":"2021-01-15T18:47:50.230770+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:50.230770+0000","last_undegraded":"2021-01-15T18:47:50.230770+0000","last_fullsized":"2021-01-15T18:47:50.230770+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7697,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17457318,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7462","reported_seq":"7679","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.157961+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:50.157961+0000","last_peered":"2021-01-15T18:47:50.157961+0000","last_clean":"2021-01-15T18:47:50.157961+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:50.157961+0000","last_undegraded":"2021-01-15T18:47:50.157961+0000","last_fullsized":"2021-01-15T18:47:50.157961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7462,"ondisk_log_size":7462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16908454,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7538","reported_seq":"9655","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:50.465896+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:50.465896+0000","last_peered":"2021-01-15T18:47:50.465896+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:50.465896+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7538,"ondisk_log_size":7538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17916070,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7644","reported_seq":"7808","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:48.767069+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:48.767069+0000","last_peered":"2021-01-15T18:47:48.767069+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:48.767069+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7644,"ondisk_log_size":7644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18124883,"num_objects":2213,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7643,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":800}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7634","reported_seq":"10577","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.479377+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:50.479377+0000","last_peered":"2021-01-15T18:47:50.479377+0000","last_clean":"2021-01-15T18:47:50.479377+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:50.479377+0000","last_undegraded":"2021-01-15T18:47:50.479377+0000","last_fullsized":"2021-01-15T18:47:50.479377+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7634,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17158227,"num_objects":2095,"num_object_clones":0,"num_object_copies":8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7648","reported_seq":"10597","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.426754+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:50.426754+0000","last_peered":"2021-01-15T18:47:50.426754+0000","last_clean":"2021-01-15T18:47:50.426754+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:50.426754+0000","last_undegraded":"2021-01-15T18:47:50.426754+0000","last_fullsized":"2021-01-15T18:47:50.426754+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7648,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17281273,"num_objects":2111,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7422","reported_seq":"7670","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:50.337732+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:50.337732+0000","last_peered":"2021-01-15T18:47:50.337732+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:50.337732+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7422,"ondisk_log_size":7422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17465510,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7420,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7414","reported_seq":"10251","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.407573+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:50.407573+0000","last_peered":"2021-01-15T18:47:50.407573+0000","last_clean":"2021-01-15T18:47:50.407573+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:50.407573+0000","last_undegraded":"2021-01-15T18:47:50.407573+0000","last_fullsized":"2021-01-15T18:47:50.407573+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7414,"ondisk_log_size":7414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17637542,"num_objects":2154,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7359","reported_seq":"10262","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:49.506679+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:49.506679+0000","last_peered":"2021-01-15T18:47:49.506679+0000","last_clean":"2021-01-15T18:47:49.506679+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:49.506679+0000","last_undegraded":"2021-01-15T18:47:49.506679+0000","last_fullsized":"2021-01-15T18:47:49.506679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7359,"ondisk_log_size":7359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17334438,"num_objects":2117,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7755","reported_seq":"7989","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:50.368882+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:50.368882+0000","last_peered":"2021-01-15T18:47:50.368882+0000","last_clean":"2021-01-15T18:47:50.368882+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:50.368882+0000","last_undegraded":"2021-01-15T18:47:50.368882+0000","last_fullsized":"2021-01-15T18:47:50.368882+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7755,"ondisk_log_size":7755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17457318,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7753,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7874","reported_seq":"8148","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:50.435607+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:50.435607+0000","last_peered":"2021-01-15T18:47:50.435607+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:50.435607+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7874,"ondisk_log_size":7874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17760588,"num_objects":2170,"num_object_clones":0,"num_object_copies":8680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7440","reported_seq":"10313","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:48.769905+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:48.769905+0000","last_peered":"2021-01-15T18:47:48.769905+0000","last_clean":"2021-01-15T18:47:48.769905+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:48.769905+0000","last_undegraded":"2021-01-15T18:47:48.769905+0000","last_fullsized":"2021-01-15T18:47:48.769905+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7440,"ondisk_log_size":7440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17195174,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":278489388,"num_objects":34010,"num_object_clones":0,"num_object_copies":136040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120715,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14358,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":558047232,"data_stored":558047232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120742,"ondisk_log_size":120742,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372276,"kb_used_data":136556,"kb_used_omap":0,"kb_used_meta":235711,"kb_avail":93999564,"statfs":{"total":96636764160,"available":96255553536,"internally_reserved":0,"allocated":139833344,"data_stored":139580166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":241368116},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":2.7989999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.99199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":3.6859999999999999},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":6.4379999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364976,"kb_used_data":137940,"kb_used_omap":0,"kb_used_meta":227008,"kb_avail":94006864,"statfs":{"total":96636764160,"available":96263028736,"internally_reserved":0,"allocated":141250560,"data_stored":140997382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":232456192},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.7539999999999996},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":0.433}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":24.071000000000002},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":6.5129999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":0.93300000000000005}]}]},{"osd":0,"up_from":22,"seq":94489280616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377040,"kb_used_data":137600,"kb_used_omap":0,"kb_used_meta":239359,"kb_avail":93994800,"statfs":{"total":96636764160,"available":96250675200,"internally_reserved":0,"allocated":140902400,"data_stored":140649222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245103668},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":4.0469999999999997},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":3.6659999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":0.436},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.52600000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.80900000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366160,"kb_used_data":136776,"kb_used_omap":0,"kb_used_meta":229311,"kb_avail":94005680,"statfs":{"total":96636764160,"available":96261816320,"internally_reserved":0,"allocated":140058624,"data_stored":139805446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234814516},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,0,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.61299999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.877}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0820000000000001}]}]}]}} 2021-01-15T18:47:54.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:47:55.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:55.678 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7049/135324 objects degraded (5.209%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":526,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33831,"data_bytes":277023020,"bytes_used":1522118656,"bytes_avail":385024937984,"bytes_total":386547056640,"degraded_objects":7049,"degraded_total":135324,"degraded_ratio":0.052089799296503217,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:47:55.735 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:47:55.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:47:56.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:47:56.813 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:47:56.856 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2021-01-15T18:47:56.604987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":270403884,"num_objects":33023,"num_object_clones":0,"num_object_copies":132092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121702,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14410,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121729,"ondisk_log_size":121729,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1450736,"kb_used_data":531548,"kb_used_omap":2,"kb_used_meta":918973,"kb_avail":376036624,"statfs":{"total":386547056640,"available":385061502976,"internally_reserved":0,"allocated":544305152,"data_stored":543292440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":941028508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,8,11,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":123,"apply_latency_ms":123,"commit_latency_ns":123000000,"apply_latency_ns":123000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23633920,"num_objects":-2885,"num_object_clones":0,"num_object_copies":-11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005177"},"pg_stats":[{"pgid":"1.f","version":"48'7417","reported_seq":"7683","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:55.273421+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:47:55.273421+0000","last_peered":"2021-01-15T18:47:55.273421+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:47:55.273421+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7417,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17129472,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7417,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7482","reported_seq":"10414","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.388912+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:47:55.388912+0000","last_peered":"2021-01-15T18:47:55.388912+0000","last_clean":"2021-01-15T18:47:55.388912+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:47:55.388912+0000","last_undegraded":"2021-01-15T18:47:55.388912+0000","last_fullsized":"2021-01-15T18:47:55.388912+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7482,"ondisk_log_size":7482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16212134,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7480,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7479","reported_seq":"10418","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:53.306907+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:53.306907+0000","last_peered":"2021-01-15T18:47:53.306907+0000","last_clean":"2021-01-15T18:47:53.306907+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:53.306907+0000","last_undegraded":"2021-01-15T18:47:53.306907+0000","last_fullsized":"2021-01-15T18:47:53.306907+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7479,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16486507,"num_objects":2014,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7478,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7709","reported_seq":"7963","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:55.388670+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:47:55.388670+0000","last_peered":"2021-01-15T18:47:55.388670+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:47:55.388670+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7709,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17063936,"num_objects":2083,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7757","reported_seq":"8357","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.353331+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:47:55.353331+0000","last_peered":"2021-01-15T18:47:55.353331+0000","last_clean":"2021-01-15T18:47:55.353331+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:47:55.353331+0000","last_undegraded":"2021-01-15T18:47:55.353331+0000","last_fullsized":"2021-01-15T18:47:55.353331+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7757,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16965798,"num_objects":2072,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7520","reported_seq":"7739","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.335655+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:47:55.335655+0000","last_peered":"2021-01-15T18:47:55.335655+0000","last_clean":"2021-01-15T18:47:55.335655+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:47:55.335655+0000","last_undegraded":"2021-01-15T18:47:55.335655+0000","last_fullsized":"2021-01-15T18:47:55.335655+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7520,"ondisk_log_size":7520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16433318,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7518,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7610","reported_seq":"9831","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:55.470283+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:47:55.470283+0000","last_peered":"2021-01-15T18:47:55.470283+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:47:55.470283+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7610,"ondisk_log_size":7610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17326246,"num_objects":2116,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1079,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1799}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7709","reported_seq":"7873","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:53.882808+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:53.882808+0000","last_peered":"2021-01-15T18:47:53.882808+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:53.882808+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7709,"ondisk_log_size":7709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17592403,"num_objects":2148,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7701","reported_seq":"10644","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.326564+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:47:55.326564+0000","last_peered":"2021-01-15T18:47:55.326564+0000","last_clean":"2021-01-15T18:47:55.326564+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:47:55.326564+0000","last_undegraded":"2021-01-15T18:47:55.326564+0000","last_fullsized":"2021-01-15T18:47:55.326564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7701,"ondisk_log_size":7701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16609363,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7709","reported_seq":"10658","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.343821+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:47:55.343821+0000","last_peered":"2021-01-15T18:47:55.343821+0000","last_clean":"2021-01-15T18:47:55.343821+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:47:55.343821+0000","last_undegraded":"2021-01-15T18:47:55.343821+0000","last_fullsized":"2021-01-15T18:47:55.343821+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7709,"ondisk_log_size":7709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16781561,"num_objects":2050,"num_object_clones":0,"num_object_copies":8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7706,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7489","reported_seq":"7735","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:55.388434+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:47:55.388434+0000","last_peered":"2021-01-15T18:47:55.388434+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:47:55.388434+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7489,"ondisk_log_size":7489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16916646,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7475","reported_seq":"10312","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.066032+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:47:55.066032+0000","last_peered":"2021-01-15T18:47:55.066032+0000","last_clean":"2021-01-15T18:47:55.066032+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:47:55.066032+0000","last_undegraded":"2021-01-15T18:47:55.066032+0000","last_fullsized":"2021-01-15T18:47:55.066032+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7475,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17137830,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7414","reported_seq":"10317","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:54.440747+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:54.440747+0000","last_peered":"2021-01-15T18:47:54.440747+0000","last_clean":"2021-01-15T18:47:54.440747+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:54.440747+0000","last_undegraded":"2021-01-15T18:47:54.440747+0000","last_fullsized":"2021-01-15T18:47:54.440747+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7414,"ondisk_log_size":7414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16883878,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7817","reported_seq":"8051","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:55.339881+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:47:55.339881+0000","last_peered":"2021-01-15T18:47:55.339881+0000","last_clean":"2021-01-15T18:47:55.339881+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:47:55.339881+0000","last_undegraded":"2021-01-15T18:47:55.339881+0000","last_fullsized":"2021-01-15T18:47:55.339881+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7817,"ondisk_log_size":7817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16949414,"num_objects":2070,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7945","reported_seq":"8217","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:55.428071+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:47:55.428071+0000","last_peered":"2021-01-15T18:47:55.428071+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:47:55.428071+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7945,"ondisk_log_size":7945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17178956,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":714}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7496","reported_seq":"10369","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:53.728280+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:53.728280+0000","last_peered":"2021-01-15T18:47:53.728280+0000","last_clean":"2021-01-15T18:47:53.728280+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:53.728280+0000","last_undegraded":"2021-01-15T18:47:53.728280+0000","last_fullsized":"2021-01-15T18:47:53.728280+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7496,"ondisk_log_size":7496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16736422,"num_objects":2044,"num_object_clones":0,"num_object_copies":8176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":270403884,"num_objects":33023,"num_object_clones":0,"num_object_copies":132092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121702,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14410,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":541466624,"data_stored":541466624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121729,"ondisk_log_size":121729,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215205,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361868,"kb_used_data":132404,"kb_used_omap":0,"kb_used_meta":229439,"kb_avail":94009972,"statfs":{"total":96636764160,"available":96266211328,"internally_reserved":0,"allocated":135581696,"data_stored":135328518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234945588},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.99199999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.099}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":0.746},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":0.88400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370968,"kb_used_data":133796,"kb_used_omap":0,"kb_used_meta":237120,"kb_avail":94000872,"statfs":{"total":96636764160,"available":96256892928,"internally_reserved":0,"allocated":137007104,"data_stored":136753926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242810880},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":3.133,"15min":3.5939999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.33800000000000002},"max":{"1min":11.816000000000001,"5min":27.870000000000001,"15min":36.380000000000003},"last":2.8879999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.6349999999999998,"15min":1.675},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.6189999999999998,"5min":23.98,"15min":23.98},"last":2.7879999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":4.3760000000000003},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":3.8010000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":4.258,"5min":3.5819999999999999,"15min":2.6789999999999998},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":25.335000000000001,"5min":25.335000000000001,"15min":25.335000000000001},"last":2.9940000000000002},{"interface":"front","average":{"1min":1.6970000000000001,"5min":3.0899999999999999,"15min":2.5369999999999999},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5039999999999996,"5min":18.34,"15min":18.34},"last":3.101}]}]},{"osd":0,"up_from":22,"seq":94489280617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366232,"kb_used_data":132764,"kb_used_omap":0,"kb_used_meta":233407,"kb_avail":94005608,"statfs":{"total":96636764160,"available":96261742592,"internally_reserved":0,"allocated":135950336,"data_stored":135697158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":239008820},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.54500000000000004},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:46:57 2021","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":3.1160000000000001,"15min":2.649},"min":{"1min":0.435,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":11.878,"5min":15.74,"15min":27.77},"last":0.45100000000000001},{"interface":"front","average":{"1min":3.1440000000000001,"5min":2.9319999999999999,"15min":2.673},"min":{"1min":0.55700000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":12.413,"5min":12.413,"15min":22.765000000000001},"last":0.57499999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.64000000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351668,"kb_used_data":132584,"kb_used_omap":0,"kb_used_meta":219007,"kb_avail":94020172,"statfs":{"total":96636764160,"available":96276656128,"internally_reserved":0,"allocated":135766016,"data_stored":135512838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":224263220},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":19.954000000000001},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":19.056999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":3.8330000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.498},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.65900000000000003}]}]}]}} 2021-01-15T18:47:59.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:01.106 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:01.111 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:01.141 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2021-01-15T18:48:00.605881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262056236,"num_objects":32004,"num_object_clones":0,"num_object_copies":128016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14475,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122748,"ondisk_log_size":122748,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1471112,"kb_used_data":516108,"kb_used_omap":2,"kb_used_meta":954877,"kb_avail":376016248,"statfs":{"total":386547056640,"available":385040637952,"internally_reserved":0,"allocated":528494592,"data_stored":527481880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":977794204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,9,11,12,6,9,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24879104,"num_objects":-3037,"num_object_clones":0,"num_object_copies":-12148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"1.f","version":"48'7477","reported_seq":"7743","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.416232+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:00.416232+0000","last_peered":"2021-01-15T18:48:00.416232+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:00.416232+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7477,"ondisk_log_size":7477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16637952,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7542","reported_seq":"10474","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.319009+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:00.319009+0000","last_peered":"2021-01-15T18:48:00.319009+0000","last_clean":"2021-01-15T18:48:00.319009+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:00.319009+0000","last_undegraded":"2021-01-15T18:48:00.319009+0000","last_fullsized":"2021-01-15T18:48:00.319009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7542,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15720614,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7549","reported_seq":"10488","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:58.923946+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:58.923946+0000","last_peered":"2021-01-15T18:47:58.923946+0000","last_clean":"2021-01-15T18:47:58.923946+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:58.923946+0000","last_undegraded":"2021-01-15T18:47:58.923946+0000","last_fullsized":"2021-01-15T18:47:58.923946+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7549,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15913067,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7548,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7776","reported_seq":"8030","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.422354+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:00.422354+0000","last_peered":"2021-01-15T18:48:00.422354+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:00.422354+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7776,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7776,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7819","reported_seq":"8419","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.281145+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:00.281145+0000","last_peered":"2021-01-15T18:48:00.281145+0000","last_clean":"2021-01-15T18:48:00.281145+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:00.281145+0000","last_undegraded":"2021-01-15T18:48:00.281145+0000","last_fullsized":"2021-01-15T18:48:00.281145+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7819,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16457894,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7580","reported_seq":"7799","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.449986+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:00.449986+0000","last_peered":"2021-01-15T18:48:00.449986+0000","last_clean":"2021-01-15T18:48:00.449986+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:00.449986+0000","last_undegraded":"2021-01-15T18:48:00.449986+0000","last_fullsized":"2021-01-15T18:48:00.449986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7580,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15941798,"num_objects":1947,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7667","reported_seq":"10017","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.503830+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:00.503830+0000","last_peered":"2021-01-15T18:48:00.503830+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:00.503830+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7667,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16859302,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7763","reported_seq":"7927","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:58.885891+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:58.885891+0000","last_peered":"2021-01-15T18:47:58.885891+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:58.885891+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7763,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17150035,"num_objects":2094,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7766","reported_seq":"10709","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.332984+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:00.332984+0000","last_peered":"2021-01-15T18:48:00.332984+0000","last_clean":"2021-01-15T18:48:00.332984+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:00.332984+0000","last_undegraded":"2021-01-15T18:48:00.332984+0000","last_fullsized":"2021-01-15T18:48:00.332984+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7766,"ondisk_log_size":7766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16076883,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7765,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7772","reported_seq":"10721","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.400607+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:00.400607+0000","last_peered":"2021-01-15T18:48:00.400607+0000","last_clean":"2021-01-15T18:48:00.400607+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:00.400607+0000","last_undegraded":"2021-01-15T18:48:00.400607+0000","last_fullsized":"2021-01-15T18:48:00.400607+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7772,"ondisk_log_size":7772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16265465,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7554","reported_seq":"7801","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.458915+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:00.458915+0000","last_peered":"2021-01-15T18:48:00.458915+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:00.458915+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7554,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16384166,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7552,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7553","reported_seq":"10390","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.458093+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:00.458093+0000","last_peered":"2021-01-15T18:48:00.458093+0000","last_clean":"2021-01-15T18:48:00.458093+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:00.458093+0000","last_undegraded":"2021-01-15T18:48:00.458093+0000","last_fullsized":"2021-01-15T18:48:00.458093+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7553,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16498854,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7483","reported_seq":"10383","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:59.657269+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:59.657269+0000","last_peered":"2021-01-15T18:47:59.657269+0000","last_clean":"2021-01-15T18:47:59.657269+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:59.657269+0000","last_undegraded":"2021-01-15T18:47:59.657269+0000","last_fullsized":"2021-01-15T18:47:59.657269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7483,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16318630,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7881","reported_seq":"8114","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.461279+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:00.461279+0000","last_peered":"2021-01-15T18:48:00.461279+0000","last_clean":"2021-01-15T18:48:00.461279+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:00.461279+0000","last_undegraded":"2021-01-15T18:48:00.461279+0000","last_fullsized":"2021-01-15T18:48:00.461279+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7881,"ondisk_log_size":7881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16425126,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7995","reported_seq":"8269","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.457840+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:00.457840+0000","last_peered":"2021-01-15T18:48:00.457840+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:00.457840+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7995,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16769356,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7571","reported_seq":"10444","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:58.680902+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:58.680902+0000","last_peered":"2021-01-15T18:47:58.680902+0000","last_clean":"2021-01-15T18:47:58.680902+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:58.680902+0000","last_undegraded":"2021-01-15T18:47:58.680902+0000","last_fullsized":"2021-01-15T18:47:58.680902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7571,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16122022,"num_objects":1969,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":262056236,"num_objects":32004,"num_object_clones":0,"num_object_copies":128016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14475,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525041664,"data_stored":525041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122748,"ondisk_log_size":122748,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367912,"kb_used_data":128428,"kb_used_omap":0,"kb_used_meta":239423,"kb_avail":94003928,"statfs":{"total":96636764160,"available":96260022272,"internally_reserved":0,"allocated":131510272,"data_stored":131257094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245169204},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,0,0,9,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":18.241}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":23.808},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":23.907}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":24.103000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372796,"kb_used_data":129568,"kb_used_omap":0,"kb_used_meta":243200,"kb_avail":93999044,"statfs":{"total":96636764160,"available":96255021056,"internally_reserved":0,"allocated":132677632,"data_stored":132424454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":249036800},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.087},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":5.5880000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.6539999999999999},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":4.548}]}]},{"osd":0,"up_from":22,"seq":94489280618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372628,"kb_used_data":129464,"kb_used_omap":0,"kb_used_meta":243135,"kb_avail":93999212,"statfs":{"total":96636764160,"available":96255193088,"internally_reserved":0,"allocated":132571136,"data_stored":132317958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":248970292},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.40200000000000002},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.63}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.753}]}]},{"osd":1,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357776,"kb_used_data":128648,"kb_used_omap":0,"kb_used_meta":229119,"kb_avail":94014064,"statfs":{"total":96636764160,"available":96270401536,"internally_reserved":0,"allocated":131735552,"data_stored":131482374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234617908},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.1839999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.81000000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":4.7190000000000003},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":4.8179999999999996}]}]}]}} 2021-01-15T18:48:01.143 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:02.536 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:02.537 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6932/128016 objects degraded (5.415%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":533,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32004,"data_bytes":262056236,"bytes_used":1506418688,"bytes_avail":385040637952,"bytes_total":386547056640,"degraded_objects":6932,"degraded_total":128016,"degraded_ratio":0.054149481314835643,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:48:02.577 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:02.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:03.673 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:03.714 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:03.732 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2021-01-15T18:48:02.606351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262056236,"num_objects":32004,"num_object_clones":0,"num_object_copies":128016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14475,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122748,"ondisk_log_size":122748,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1471112,"kb_used_data":516108,"kb_used_omap":2,"kb_used_meta":954877,"kb_avail":376016248,"statfs":{"total":386547056640,"available":385040637952,"internally_reserved":0,"allocated":528494592,"data_stored":527481880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":977794204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,9,11,12,6,9,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16433152,"num_objects":-2006,"num_object_clones":0,"num_object_copies":-8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2006,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002598"},"pg_stats":[{"pgid":"1.f","version":"48'7477","reported_seq":"7743","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.416232+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:00.416232+0000","last_peered":"2021-01-15T18:48:00.416232+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:00.416232+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7477,"ondisk_log_size":7477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16637952,"num_objects":2031,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":781}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7542","reported_seq":"10474","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.319009+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:00.319009+0000","last_peered":"2021-01-15T18:48:00.319009+0000","last_clean":"2021-01-15T18:48:00.319009+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:00.319009+0000","last_undegraded":"2021-01-15T18:48:00.319009+0000","last_fullsized":"2021-01-15T18:48:00.319009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7542,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15720614,"num_objects":1920,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7549","reported_seq":"10488","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:58.923946+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:47:58.923946+0000","last_peered":"2021-01-15T18:47:58.923946+0000","last_clean":"2021-01-15T18:47:58.923946+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:47:58.923946+0000","last_undegraded":"2021-01-15T18:47:58.923946+0000","last_fullsized":"2021-01-15T18:47:58.923946+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7549,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15913067,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7548,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7776","reported_seq":"8030","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.422354+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:00.422354+0000","last_peered":"2021-01-15T18:48:00.422354+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:00.422354+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7776,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7776,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7819","reported_seq":"8419","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.281145+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:00.281145+0000","last_peered":"2021-01-15T18:48:00.281145+0000","last_clean":"2021-01-15T18:48:00.281145+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:00.281145+0000","last_undegraded":"2021-01-15T18:48:00.281145+0000","last_fullsized":"2021-01-15T18:48:00.281145+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7819,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16457894,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7580","reported_seq":"7799","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.449986+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:00.449986+0000","last_peered":"2021-01-15T18:48:00.449986+0000","last_clean":"2021-01-15T18:48:00.449986+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:00.449986+0000","last_undegraded":"2021-01-15T18:48:00.449986+0000","last_fullsized":"2021-01-15T18:48:00.449986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7580,"ondisk_log_size":7580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15941798,"num_objects":1947,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7667","reported_seq":"10017","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.503830+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:00.503830+0000","last_peered":"2021-01-15T18:48:00.503830+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:00.503830+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7667,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16859302,"num_objects":2059,"num_object_clones":0,"num_object_copies":8236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7763","reported_seq":"7927","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:47:58.885891+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:47:58.885891+0000","last_peered":"2021-01-15T18:47:58.885891+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:47:58.885891+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7763,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17150035,"num_objects":2094,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7766","reported_seq":"10709","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.332984+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:00.332984+0000","last_peered":"2021-01-15T18:48:00.332984+0000","last_clean":"2021-01-15T18:48:00.332984+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:00.332984+0000","last_undegraded":"2021-01-15T18:48:00.332984+0000","last_fullsized":"2021-01-15T18:48:00.332984+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7766,"ondisk_log_size":7766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16076883,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7765,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7772","reported_seq":"10721","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.400607+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:00.400607+0000","last_peered":"2021-01-15T18:48:00.400607+0000","last_clean":"2021-01-15T18:48:00.400607+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:00.400607+0000","last_undegraded":"2021-01-15T18:48:00.400607+0000","last_fullsized":"2021-01-15T18:48:00.400607+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7772,"ondisk_log_size":7772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16265465,"num_objects":1987,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7554","reported_seq":"7801","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.458915+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:00.458915+0000","last_peered":"2021-01-15T18:48:00.458915+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:00.458915+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7554,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16384166,"num_objects":2001,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7552,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7553","reported_seq":"10390","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.458093+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:00.458093+0000","last_peered":"2021-01-15T18:48:00.458093+0000","last_clean":"2021-01-15T18:48:00.458093+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:00.458093+0000","last_undegraded":"2021-01-15T18:48:00.458093+0000","last_fullsized":"2021-01-15T18:48:00.458093+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7553,"ondisk_log_size":7553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16498854,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7483","reported_seq":"10383","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:59.657269+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:47:59.657269+0000","last_peered":"2021-01-15T18:47:59.657269+0000","last_clean":"2021-01-15T18:47:59.657269+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:47:59.657269+0000","last_undegraded":"2021-01-15T18:47:59.657269+0000","last_fullsized":"2021-01-15T18:47:59.657269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7483,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16318630,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7881","reported_seq":"8114","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:00.461279+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:00.461279+0000","last_peered":"2021-01-15T18:48:00.461279+0000","last_clean":"2021-01-15T18:48:00.461279+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:00.461279+0000","last_undegraded":"2021-01-15T18:48:00.461279+0000","last_fullsized":"2021-01-15T18:48:00.461279+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7881,"ondisk_log_size":7881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16425126,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'7995","reported_seq":"8269","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:00.457840+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:00.457840+0000","last_peered":"2021-01-15T18:48:00.457840+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:00.457840+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7995,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16769356,"num_objects":2049,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7571","reported_seq":"10444","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:47:58.680902+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:47:58.680902+0000","last_peered":"2021-01-15T18:47:58.680902+0000","last_clean":"2021-01-15T18:47:58.680902+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:47:58.680902+0000","last_undegraded":"2021-01-15T18:47:58.680902+0000","last_fullsized":"2021-01-15T18:47:58.680902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7571,"ondisk_log_size":7571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16122022,"num_objects":1969,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":262056236,"num_objects":32004,"num_object_clones":0,"num_object_copies":128016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14475,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525041664,"data_stored":525041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122748,"ondisk_log_size":122748,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215206,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367912,"kb_used_data":128428,"kb_used_omap":0,"kb_used_meta":239423,"kb_avail":94003928,"statfs":{"total":96636764160,"available":96260022272,"internally_reserved":0,"allocated":131510272,"data_stored":131257094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245169204},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,0,0,0,9,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":18.241}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":23.808},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":23.907}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":4.2000000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":24.103000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372796,"kb_used_data":129568,"kb_used_omap":0,"kb_used_meta":243200,"kb_avail":93999044,"statfs":{"total":96636764160,"available":96255021056,"internally_reserved":0,"allocated":132677632,"data_stored":132424454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":249036800},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":1.087},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":5.5880000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":4.6539999999999999},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":4.548}]}]},{"osd":0,"up_from":22,"seq":94489280618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372628,"kb_used_data":129464,"kb_used_omap":0,"kb_used_meta":243135,"kb_avail":93999212,"statfs":{"total":96636764160,"available":96255193088,"internally_reserved":0,"allocated":132571136,"data_stored":132317958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":248970292},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.40200000000000002},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.63}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":0.753}]}]},{"osd":1,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357776,"kb_used_data":128648,"kb_used_omap":0,"kb_used_meta":229119,"kb_avail":94014064,"statfs":{"total":96636764160,"available":96270401536,"internally_reserved":0,"allocated":131735552,"data_stored":131482374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234617908},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,5,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.1839999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.81000000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":4.7190000000000003},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":4.8179999999999996}]}]}]}} 2021-01-15T18:48:06.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:07.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:07.841 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:07.864 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2021-01-15T18:48:06.607198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253511980,"num_objects":30961,"num_object_clones":0,"num_object_copies":123844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123764,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14525,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123791,"ondisk_log_size":123791,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1437472,"kb_used_data":499360,"kb_used_omap":2,"kb_used_meta":937917,"kb_avail":376049888,"statfs":{"total":386547056640,"available":385075085312,"internally_reserved":0,"allocated":511344640,"data_stored":510331928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":960427164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,14,5,23,16,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":133,"apply_latency_ms":133,"commit_latency_ns":133000000,"apply_latency_ns":133000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23511040,"num_objects":-2870,"num_object_clones":0,"num_object_copies":-11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002602"},"pg_stats":[{"pgid":"1.f","version":"48'7538","reported_seq":"7803","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.374009+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:05.374009+0000","last_peered":"2021-01-15T18:48:05.374009+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:05.374009+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7538,"ondisk_log_size":7538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16138240,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7538,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7613","reported_seq":"10545","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.296520+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:05.296520+0000","last_peered":"2021-01-15T18:48:05.296520+0000","last_clean":"2021-01-15T18:48:05.296520+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:05.296520+0000","last_undegraded":"2021-01-15T18:48:05.296520+0000","last_fullsized":"2021-01-15T18:48:05.296520+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7613,"ondisk_log_size":7613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15138982,"num_objects":1849,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7611,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7608","reported_seq":"10547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:03.997421+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:03.997421+0000","last_peered":"2021-01-15T18:48:03.997421+0000","last_clean":"2021-01-15T18:48:03.997421+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:03.997421+0000","last_undegraded":"2021-01-15T18:48:03.997421+0000","last_fullsized":"2021-01-15T18:48:03.997421+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7608,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15429739,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7607,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7844","reported_seq":"8098","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.421112+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:05.421112+0000","last_peered":"2021-01-15T18:48:05.421112+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:05.421112+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15958016,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7889","reported_seq":"8487","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.504297+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:05.504297+0000","last_peered":"2021-01-15T18:48:05.504297+0000","last_clean":"2021-01-15T18:48:05.504297+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:05.504297+0000","last_undegraded":"2021-01-15T18:48:05.504297+0000","last_fullsized":"2021-01-15T18:48:05.504297+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7889,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15884454,"num_objects":1940,"num_object_clones":0,"num_object_copies":7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7636","reported_seq":"7855","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.504410+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:05.504410+0000","last_peered":"2021-01-15T18:48:05.504410+0000","last_clean":"2021-01-15T18:48:05.504410+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:05.504410+0000","last_undegraded":"2021-01-15T18:48:05.504410+0000","last_fullsized":"2021-01-15T18:48:05.504410+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7636,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15483046,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7733","reported_seq":"10183","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.511678+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:05.511678+0000","last_peered":"2021-01-15T18:48:05.511678+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:05.511678+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7733,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16318630,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7821","reported_seq":"7985","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:03.927048+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:03.927048+0000","last_peered":"2021-01-15T18:48:03.927048+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:03.927048+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7821,"ondisk_log_size":7821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16674899,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7820,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":623}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7847","reported_seq":"10790","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.329599+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:05.329599+0000","last_peered":"2021-01-15T18:48:05.329599+0000","last_clean":"2021-01-15T18:48:05.329599+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:05.329599+0000","last_undegraded":"2021-01-15T18:48:05.329599+0000","last_fullsized":"2021-01-15T18:48:05.329599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7847,"ondisk_log_size":7847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15413331,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7830","reported_seq":"10779","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.283798+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:05.283798+0000","last_peered":"2021-01-15T18:48:05.283798+0000","last_clean":"2021-01-15T18:48:05.283798+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:05.283798+0000","last_undegraded":"2021-01-15T18:48:05.283798+0000","last_fullsized":"2021-01-15T18:48:05.283798+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7830,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15790329,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7827,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7620","reported_seq":"7868","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.462545+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:05.462545+0000","last_peered":"2021-01-15T18:48:05.462545+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:05.462545+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7620,"ondisk_log_size":7620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15843494,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7624","reported_seq":"10460","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.528891+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:05.528891+0000","last_peered":"2021-01-15T18:48:05.528891+0000","last_clean":"2021-01-15T18:48:05.528891+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:05.528891+0000","last_undegraded":"2021-01-15T18:48:05.528891+0000","last_fullsized":"2021-01-15T18:48:05.528891+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7624,"ondisk_log_size":7624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15917222,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7622,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7550","reported_seq":"10453","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:04.605170+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:04.605170+0000","last_peered":"2021-01-15T18:48:04.605170+0000","last_clean":"2021-01-15T18:48:04.605170+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:04.605170+0000","last_undegraded":"2021-01-15T18:48:04.605170+0000","last_fullsized":"2021-01-15T18:48:04.605170+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7550,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15769766,"num_objects":1926,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7957","reported_seq":"8190","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.189810+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:05.189810+0000","last_peered":"2021-01-15T18:48:05.189810+0000","last_clean":"2021-01-15T18:48:05.189810+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:05.189810+0000","last_undegraded":"2021-01-15T18:48:05.189810+0000","last_fullsized":"2021-01-15T18:48:05.189810+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7957,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15802534,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7955,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8065","reported_seq":"8339","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.388898+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:05.388898+0000","last_peered":"2021-01-15T18:48:05.388898+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:05.388898+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8065,"ondisk_log_size":8065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16195916,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7616","reported_seq":"10489","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:03.638329+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:03.638329+0000","last_peered":"2021-01-15T18:48:03.638329+0000","last_clean":"2021-01-15T18:48:03.638329+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:03.638329+0000","last_undegraded":"2021-01-15T18:48:03.638329+0000","last_fullsized":"2021-01-15T18:48:03.638329+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7616,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15753382,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":253511980,"num_objects":30961,"num_object_clones":0,"num_object_copies":123844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123764,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14525,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507805696,"data_stored":507805696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123791,"ondisk_log_size":123791,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369660,"kb_used_data":124152,"kb_used_omap":0,"kb_used_meta":245439,"kb_avail":94002180,"statfs":{"total":96636764160,"available":96258232320,"internally_reserved":0,"allocated":127131648,"data_stored":126878470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":251329588},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,10,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.8399999999999999},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":4.9740000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":3.153}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362324,"kb_used_data":125440,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":94009516,"statfs":{"total":96636764160,"available":96265744384,"internally_reserved":0,"allocated":128450560,"data_stored":128197382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67900000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":2.641},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":3.218},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.85999999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358112,"kb_used_data":125364,"kb_used_omap":0,"kb_used_meta":232703,"kb_avail":94013728,"statfs":{"total":96636764160,"available":96270057472,"internally_reserved":0,"allocated":128372736,"data_stored":128119558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":238287924},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":4.3760000000000003},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":4.4779999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":4.0270000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":3.2229999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":12.444000000000001},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":12.558999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347376,"kb_used_data":124404,"kb_used_omap":0,"kb_used_meta":222911,"kb_avail":94024464,"statfs":{"total":96636764160,"available":96281051136,"internally_reserved":0,"allocated":127389696,"data_stored":127136518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":228260916},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":2.7559999999999998},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.874}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.0629999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.96799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.0270000000000001}]}]}]}} 2021-01-15T18:48:07.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:09.031 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:09.031 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6882/123844 objects degraded (5.557%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30961,"data_bytes":253511980,"bytes_used":1471971328,"bytes_avail":385075085312,"bytes_total":386547056640,"degraded_objects":6882,"degraded_total":123844,"degraded_ratio":0.055569910532605536,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2021-01-15T18:45:24.541155+0000","services":{}},"progress_events":{}} 2021-01-15T18:48:09.130 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:09.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:10.320 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:10.344 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:10.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2021-01-15T18:48:08.607663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253511980,"num_objects":30961,"num_object_clones":0,"num_object_copies":123844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123764,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14525,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123791,"ondisk_log_size":123791,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1437472,"kb_used_data":499360,"kb_used_omap":2,"kb_used_meta":937917,"kb_avail":376049888,"statfs":{"total":386547056640,"available":385075085312,"internally_reserved":0,"allocated":511344640,"data_stored":510331928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":960427164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,14,5,23,16,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":133,"apply_latency_ms":133,"commit_latency_ns":133000000,"apply_latency_ns":133000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16891904,"num_objects":-2062,"num_object_clones":0,"num_object_copies":-8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"1.f","version":"48'7538","reported_seq":"7803","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.374009+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:05.374009+0000","last_peered":"2021-01-15T18:48:05.374009+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:05.374009+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7538,"ondisk_log_size":7538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16138240,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7538,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7613","reported_seq":"10545","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.296520+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:05.296520+0000","last_peered":"2021-01-15T18:48:05.296520+0000","last_clean":"2021-01-15T18:48:05.296520+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:05.296520+0000","last_undegraded":"2021-01-15T18:48:05.296520+0000","last_fullsized":"2021-01-15T18:48:05.296520+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7613,"ondisk_log_size":7613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15138982,"num_objects":1849,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7611,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7608","reported_seq":"10547","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:03.997421+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:03.997421+0000","last_peered":"2021-01-15T18:48:03.997421+0000","last_clean":"2021-01-15T18:48:03.997421+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:03.997421+0000","last_undegraded":"2021-01-15T18:48:03.997421+0000","last_fullsized":"2021-01-15T18:48:03.997421+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7608,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15429739,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7607,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7844","reported_seq":"8098","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.421112+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:05.421112+0000","last_peered":"2021-01-15T18:48:05.421112+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:05.421112+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15958016,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7889","reported_seq":"8487","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.504297+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:05.504297+0000","last_peered":"2021-01-15T18:48:05.504297+0000","last_clean":"2021-01-15T18:48:05.504297+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:05.504297+0000","last_undegraded":"2021-01-15T18:48:05.504297+0000","last_fullsized":"2021-01-15T18:48:05.504297+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7889,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15884454,"num_objects":1940,"num_object_clones":0,"num_object_copies":7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7636","reported_seq":"7855","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.504410+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:05.504410+0000","last_peered":"2021-01-15T18:48:05.504410+0000","last_clean":"2021-01-15T18:48:05.504410+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:05.504410+0000","last_undegraded":"2021-01-15T18:48:05.504410+0000","last_fullsized":"2021-01-15T18:48:05.504410+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7636,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15483046,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7733","reported_seq":"10183","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.511678+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:05.511678+0000","last_peered":"2021-01-15T18:48:05.511678+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:05.511678+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7733,"ondisk_log_size":7733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16318630,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7821","reported_seq":"7985","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:03.927048+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:03.927048+0000","last_peered":"2021-01-15T18:48:03.927048+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:03.927048+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7821,"ondisk_log_size":7821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16674899,"num_objects":2036,"num_object_clones":0,"num_object_copies":8144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7820,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":623}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7847","reported_seq":"10790","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.329599+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:05.329599+0000","last_peered":"2021-01-15T18:48:05.329599+0000","last_clean":"2021-01-15T18:48:05.329599+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:05.329599+0000","last_undegraded":"2021-01-15T18:48:05.329599+0000","last_fullsized":"2021-01-15T18:48:05.329599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7847,"ondisk_log_size":7847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15413331,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7830","reported_seq":"10779","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.283798+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:05.283798+0000","last_peered":"2021-01-15T18:48:05.283798+0000","last_clean":"2021-01-15T18:48:05.283798+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:05.283798+0000","last_undegraded":"2021-01-15T18:48:05.283798+0000","last_fullsized":"2021-01-15T18:48:05.283798+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7830,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15790329,"num_objects":1929,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7827,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7620","reported_seq":"7868","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.462545+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:05.462545+0000","last_peered":"2021-01-15T18:48:05.462545+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:05.462545+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7620,"ondisk_log_size":7620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15843494,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7624","reported_seq":"10460","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.528891+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:05.528891+0000","last_peered":"2021-01-15T18:48:05.528891+0000","last_clean":"2021-01-15T18:48:05.528891+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:05.528891+0000","last_undegraded":"2021-01-15T18:48:05.528891+0000","last_fullsized":"2021-01-15T18:48:05.528891+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7624,"ondisk_log_size":7624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15917222,"num_objects":1944,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7622,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7550","reported_seq":"10453","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:04.605170+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:04.605170+0000","last_peered":"2021-01-15T18:48:04.605170+0000","last_clean":"2021-01-15T18:48:04.605170+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:04.605170+0000","last_undegraded":"2021-01-15T18:48:04.605170+0000","last_fullsized":"2021-01-15T18:48:04.605170+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7550,"ondisk_log_size":7550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15769766,"num_objects":1926,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'7957","reported_seq":"8190","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:05.189810+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:05.189810+0000","last_peered":"2021-01-15T18:48:05.189810+0000","last_clean":"2021-01-15T18:48:05.189810+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:05.189810+0000","last_undegraded":"2021-01-15T18:48:05.189810+0000","last_fullsized":"2021-01-15T18:48:05.189810+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7957,"ondisk_log_size":7957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15802534,"num_objects":1930,"num_object_clones":0,"num_object_copies":7720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7955,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8065","reported_seq":"8339","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:05.388898+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:05.388898+0000","last_peered":"2021-01-15T18:48:05.388898+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:05.388898+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8065,"ondisk_log_size":8065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16195916,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7616","reported_seq":"10489","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:03.638329+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:03.638329+0000","last_peered":"2021-01-15T18:48:03.638329+0000","last_clean":"2021-01-15T18:48:03.638329+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:03.638329+0000","last_undegraded":"2021-01-15T18:48:03.638329+0000","last_fullsized":"2021-01-15T18:48:03.638329+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7616,"ondisk_log_size":7616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15753382,"num_objects":1924,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7614,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":253511980,"num_objects":30961,"num_object_clones":0,"num_object_copies":123844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123764,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14525,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507805696,"data_stored":507805696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123791,"ondisk_log_size":123791,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369660,"kb_used_data":124152,"kb_used_omap":0,"kb_used_meta":245439,"kb_avail":94002180,"statfs":{"total":96636764160,"available":96258232320,"internally_reserved":0,"allocated":127131648,"data_stored":126878470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":251329588},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,10,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":3.456,"15min":3.1579999999999999},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":24.763000000000002,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.8399999999999999},{"interface":"front","average":{"1min":1.845,"5min":3.153,"15min":3.2519999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.41299999999999998},"max":{"1min":12.958,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:47:09 2021","interfaces":[{"interface":"back","average":{"1min":3.9239999999999999,"5min":3.6120000000000001,"15min":2.6360000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":18.959,"5min":24.062000000000001,"15min":24.062000000000001},"last":4.9740000000000002},{"interface":"front","average":{"1min":3.4660000000000002,"5min":4.3239999999999998,"15min":3.3559999999999999},"min":{"1min":0.76700000000000002,"5min":0.54300000000000004,"15min":0.503},"max":{"1min":14.429,"5min":23.154,"15min":23.154},"last":3.153}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362324,"kb_used_data":125440,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":94009516,"statfs":{"total":96636764160,"available":96265744384,"internally_reserved":0,"allocated":128450560,"data_stored":128197382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67900000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":2.641},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":3.218},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.85999999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358112,"kb_used_data":125364,"kb_used_omap":0,"kb_used_meta":232703,"kb_avail":94013728,"statfs":{"total":96636764160,"available":96270057472,"internally_reserved":0,"allocated":128372736,"data_stored":128119558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":238287924},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":4.3760000000000003},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":4.4779999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":4.0270000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":3.2229999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:02 2021","interfaces":[{"interface":"back","average":{"1min":6.9630000000000001,"5min":4.6470000000000002,"15min":4.4669999999999996},"min":{"1min":0.499,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":56.615000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":12.444000000000001},{"interface":"front","average":{"1min":6.3840000000000003,"5min":5.5019999999999998,"15min":3.3119999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":24.734999999999999,"5min":56.927,"15min":56.927},"last":12.558999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347376,"kb_used_data":124404,"kb_used_omap":0,"kb_used_meta":222911,"kb_avail":94024464,"statfs":{"total":96636764160,"available":96281051136,"internally_reserved":0,"allocated":127389696,"data_stored":127136518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":228260916},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":2.7559999999999998},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.874}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.0629999999999999},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.96799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.0270000000000001}]}]}]}} 2021-01-15T18:48:13.401 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:14.452 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:14.459 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:14.504 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2021-01-15T18:48:12.608536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245131564,"num_objects":29938,"num_object_clones":0,"num_object_copies":119752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124787,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14578,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124814,"ondisk_log_size":124814,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1440452,"kb_used_data":481948,"kb_used_omap":2,"kb_used_meta":958269,"kb_avail":376046908,"statfs":{"total":386547056640,"available":385072033792,"internally_reserved":0,"allocated":493514752,"data_stored":492502040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":981267612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,7,13,10,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":129,"apply_latency_ms":129,"commit_latency_ns":129000000,"apply_latency_ns":129000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16924672,"num_objects":-2066,"num_object_clones":0,"num_object_copies":-8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2066,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002654"},"pg_stats":[{"pgid":"1.f","version":"48'7607","reported_seq":"7873","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:10.369059+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:10.369059+0000","last_peered":"2021-01-15T18:48:10.369059+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:10.369059+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7607,"ondisk_log_size":7607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15572992,"num_objects":1901,"num_object_clones":0,"num_object_copies":7604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7667","reported_seq":"10599","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.214018+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:10.214018+0000","last_peered":"2021-01-15T18:48:10.214018+0000","last_clean":"2021-01-15T18:48:10.214018+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:10.214018+0000","last_undegraded":"2021-01-15T18:48:10.214018+0000","last_fullsized":"2021-01-15T18:48:10.214018+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7667,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14696614,"num_objects":1795,"num_object_clones":0,"num_object_copies":7180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7667","reported_seq":"10606","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:08.823522+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:08.823522+0000","last_peered":"2021-01-15T18:48:08.823522+0000","last_clean":"2021-01-15T18:48:08.823522+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:08.823522+0000","last_undegraded":"2021-01-15T18:48:08.823522+0000","last_fullsized":"2021-01-15T18:48:08.823522+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7667,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14946411,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7666,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7908","reported_seq":"8162","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:10.439704+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:10.439704+0000","last_peered":"2021-01-15T18:48:10.439704+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:10.439704+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7908,"ondisk_log_size":7908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15433728,"num_objects":1884,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'7956","reported_seq":"8556","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.561152+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:10.561152+0000","last_peered":"2021-01-15T18:48:10.561152+0000","last_clean":"2021-01-15T18:48:10.561152+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:10.561152+0000","last_undegraded":"2021-01-15T18:48:10.561152+0000","last_fullsized":"2021-01-15T18:48:10.561152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7956,"ondisk_log_size":7956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15335590,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7696","reported_seq":"7915","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.434986+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:10.434986+0000","last_peered":"2021-01-15T18:48:10.434986+0000","last_clean":"2021-01-15T18:48:10.434986+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:10.434986+0000","last_undegraded":"2021-01-15T18:48:10.434986+0000","last_fullsized":"2021-01-15T18:48:10.434986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7696,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14991526,"num_objects":1831,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7694,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7791","reported_seq":"10347","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:10.535551+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:10.535551+0000","last_peered":"2021-01-15T18:48:10.535551+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:10.535551+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7791,"ondisk_log_size":7791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15843494,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7789,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1786}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7885","reported_seq":"8049","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:08.921450+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:08.921450+0000","last_peered":"2021-01-15T18:48:08.921450+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:08.921450+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7885,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16150611,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7884,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":559}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7916","reported_seq":"10859","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.465258+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:10.465258+0000","last_peered":"2021-01-15T18:48:10.465258+0000","last_clean":"2021-01-15T18:48:10.465258+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:10.465258+0000","last_undegraded":"2021-01-15T18:48:10.465258+0000","last_fullsized":"2021-01-15T18:48:10.465258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7916,"ondisk_log_size":7916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14848083,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7915,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7886","reported_seq":"10834","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.503076+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:10.503076+0000","last_peered":"2021-01-15T18:48:10.503076+0000","last_clean":"2021-01-15T18:48:10.503076+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:10.503076+0000","last_undegraded":"2021-01-15T18:48:10.503076+0000","last_fullsized":"2021-01-15T18:48:10.503076+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7886,"ondisk_log_size":7886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15331577,"num_objects":1873,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7688","reported_seq":"7935","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:10.511862+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:10.511862+0000","last_peered":"2021-01-15T18:48:10.511862+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:10.511862+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7688,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15286438,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7697","reported_seq":"10534","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.564852+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:10.564852+0000","last_peered":"2021-01-15T18:48:10.564852+0000","last_clean":"2021-01-15T18:48:10.564852+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:10.564852+0000","last_undegraded":"2021-01-15T18:48:10.564852+0000","last_fullsized":"2021-01-15T18:48:10.564852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7697,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15319206,"num_objects":1871,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7611","reported_seq":"10514","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:09.275629+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:09.275629+0000","last_peered":"2021-01-15T18:48:09.275629+0000","last_clean":"2021-01-15T18:48:09.275629+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:09.275629+0000","last_undegraded":"2021-01-15T18:48:09.275629+0000","last_fullsized":"2021-01-15T18:48:09.275629+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7611,"ondisk_log_size":7611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15270054,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8024","reported_seq":"8257","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:10.347754+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:10.347754+0000","last_peered":"2021-01-15T18:48:10.347754+0000","last_clean":"2021-01-15T18:48:10.347754+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:10.347754+0000","last_undegraded":"2021-01-15T18:48:10.347754+0000","last_fullsized":"2021-01-15T18:48:10.347754+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8024,"ondisk_log_size":8024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15253670,"num_objects":1863,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8129","reported_seq":"8402","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:10.516806+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:10.516806+0000","last_peered":"2021-01-15T18:48:10.516806+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:10.516806+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8129,"ondisk_log_size":8129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15671628,"num_objects":1915,"num_object_clones":0,"num_object_copies":7660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7686","reported_seq":"10559","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:08.823332+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:08.823332+0000","last_peered":"2021-01-15T18:48:08.823332+0000","last_clean":"2021-01-15T18:48:08.823332+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:08.823332+0000","last_undegraded":"2021-01-15T18:48:08.823332+0000","last_fullsized":"2021-01-15T18:48:08.823332+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7686,"ondisk_log_size":7686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15179942,"num_objects":1854,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7684,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":245131564,"num_objects":29938,"num_object_clones":0,"num_object_copies":119752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29938,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124787,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14578,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":490921984,"data_stored":490921984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124814,"ondisk_log_size":124814,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215208,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359272,"kb_used_data":120040,"kb_used_omap":0,"kb_used_meta":239167,"kb_avail":94012568,"statfs":{"total":96636764160,"available":96268869632,"internally_reserved":0,"allocated":122920960,"data_stored":122667782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244907060},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,5,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":9.1289999999999996},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.505}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.89900000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.3030000000000008},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.995}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368256,"kb_used_data":121208,"kb_used_omap":0,"kb_used_meta":246976,"kb_avail":94003584,"statfs":{"total":96636764160,"available":96259670016,"internally_reserved":0,"allocated":124116992,"data_stored":123863814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":252903424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67900000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":2.641},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.71799999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":3.218},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.85999999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363672,"kb_used_data":120488,"kb_used_omap":0,"kb_used_meta":243135,"kb_avail":94008168,"statfs":{"total":96636764160,"available":96264364032,"internally_reserved":0,"allocated":123379712,"data_stored":123126534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":248970292},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":1.0660000000000001},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.204}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.5},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.96199999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":1.087}]}]},{"osd":1,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349252,"kb_used_data":120212,"kb_used_omap":0,"kb_used_meta":228991,"kb_avail":94022588,"statfs":{"total":96636764160,"available":96279130112,"internally_reserved":0,"allocated":123097088,"data_stored":122843910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":234486836},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.4099999999999999},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.331},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.76300000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.456},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.579}]}]}]}} 2021-01-15T18:48:14.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:15.995 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:15.996 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6829/118908 objects degraded (5.743%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":546,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29727,"data_bytes":243403052,"bytes_used":1464266752,"bytes_avail":385082789888,"bytes_total":386547056640,"degraded_objects":6829,"degraded_total":118908,"degraded_ratio":0.05743095502405221,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:16.087 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:16.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:17.511 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:17.545 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:17.586 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2021-01-15T18:48:16.616231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":236177708,"num_objects":28845,"num_object_clones":0,"num_object_copies":115380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125880,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14627,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125907,"ondisk_log_size":125907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1431116,"kb_used_data":464784,"kb_used_omap":2,"kb_used_meta":966141,"kb_avail":376056244,"statfs":{"total":386547056640,"available":385081593856,"internally_reserved":0,"allocated":475938816,"data_stored":474926104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":989328540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,1,9,20,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":134,"apply_latency_ms":134,"commit_latency_ns":134000000,"apply_latency_ns":134000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24551424,"num_objects":-2997,"num_object_clones":0,"num_object_copies":-11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009478"},"pg_stats":[{"pgid":"1.f","version":"48'7666","reported_seq":"7930","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:15.505277+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:15.505277+0000","last_peered":"2021-01-15T18:48:15.505277+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:15.505277+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7666,"ondisk_log_size":7666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15089664,"num_objects":1842,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7719","reported_seq":"10650","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.612091+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:15.612091+0000","last_peered":"2021-01-15T18:48:15.612091+0000","last_clean":"2021-01-15T18:48:15.612091+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:15.612091+0000","last_undegraded":"2021-01-15T18:48:15.612091+0000","last_fullsized":"2021-01-15T18:48:15.612091+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7719,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14270630,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7739","reported_seq":"10677","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:13.825955+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:13.825955+0000","last_peered":"2021-01-15T18:48:13.825955+0000","last_clean":"2021-01-15T18:48:13.825955+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:13.825955+0000","last_undegraded":"2021-01-15T18:48:13.825955+0000","last_fullsized":"2021-01-15T18:48:13.825955+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7739,"ondisk_log_size":7739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14356587,"num_objects":1754,"num_object_clones":0,"num_object_copies":7016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7738,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'7973","reported_seq":"8226","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:15.511830+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:15.511830+0000","last_peered":"2021-01-15T18:48:15.511830+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:15.511830+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7973,"ondisk_log_size":7973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14901248,"num_objects":1819,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'8024","reported_seq":"8623","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.143720+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:15.143720+0000","last_peered":"2021-01-15T18:48:15.143720+0000","last_clean":"2021-01-15T18:48:15.143720+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:15.143720+0000","last_undegraded":"2021-01-15T18:48:15.143720+0000","last_fullsized":"2021-01-15T18:48:15.143720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8024,"ondisk_log_size":8024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14778534,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7768","reported_seq":"7987","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.579835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:15.579835+0000","last_peered":"2021-01-15T18:48:15.579835+0000","last_clean":"2021-01-15T18:48:15.579835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:15.579835+0000","last_undegraded":"2021-01-15T18:48:15.579835+0000","last_fullsized":"2021-01-15T18:48:15.579835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7768,"ondisk_log_size":7768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14401702,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7866","reported_seq":"10521","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:15.495411+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:15.495411+0000","last_peered":"2021-01-15T18:48:15.495411+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:15.495411+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7866,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15229094,"num_objects":1860,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'7967","reported_seq":"8131","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:13.904413+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:13.904413+0000","last_peered":"2021-01-15T18:48:13.904413+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:13.904413+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7967,"ondisk_log_size":7967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15478867,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7976","reported_seq":"10919","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.598942+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:15.598942+0000","last_peered":"2021-01-15T18:48:15.598942+0000","last_clean":"2021-01-15T18:48:15.598942+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:15.598942+0000","last_undegraded":"2021-01-15T18:48:15.598942+0000","last_fullsized":"2021-01-15T18:48:15.598942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7976,"ondisk_log_size":7976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14356563,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'7949","reported_seq":"10898","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.432022+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:15.432022+0000","last_peered":"2021-01-15T18:48:15.432022+0000","last_clean":"2021-01-15T18:48:15.432022+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:15.432022+0000","last_undegraded":"2021-01-15T18:48:15.432022+0000","last_fullsized":"2021-01-15T18:48:15.432022+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7949,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14815481,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7946,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7764","reported_seq":"8011","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:15.494287+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:15.494287+0000","last_peered":"2021-01-15T18:48:15.494287+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:15.494287+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7764,"ondisk_log_size":7764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14663846,"num_objects":1791,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7778","reported_seq":"10615","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.344402+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:15.344402+0000","last_peered":"2021-01-15T18:48:15.344402+0000","last_clean":"2021-01-15T18:48:15.344402+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:15.344402+0000","last_undegraded":"2021-01-15T18:48:15.344402+0000","last_fullsized":"2021-01-15T18:48:15.344402+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7778,"ondisk_log_size":7778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14655654,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7776,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7694","reported_seq":"10597","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:14.653514+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:14.653514+0000","last_peered":"2021-01-15T18:48:14.653514+0000","last_clean":"2021-01-15T18:48:14.653514+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:14.653514+0000","last_undegraded":"2021-01-15T18:48:14.653514+0000","last_fullsized":"2021-01-15T18:48:14.653514+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7694,"ondisk_log_size":7694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14590118,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8092","reported_seq":"8326","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.411332+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:15.411332+0000","last_peered":"2021-01-15T18:48:15.411332+0000","last_clean":"2021-01-15T18:48:15.411332+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:15.411332+0000","last_undegraded":"2021-01-15T18:48:15.411332+0000","last_fullsized":"2021-01-15T18:48:15.411332+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8092,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14696614,"num_objects":1795,"num_object_clones":0,"num_object_copies":7180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8189","reported_seq":"8463","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:15.433209+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:15.433209+0000","last_peered":"2021-01-15T18:48:15.433209+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:15.433209+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8189,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15180108,"num_objects":1855,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7743","reported_seq":"10615","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:13.974877+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:13.974877+0000","last_peered":"2021-01-15T18:48:13.974877+0000","last_clean":"2021-01-15T18:48:13.974877+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:13.974877+0000","last_undegraded":"2021-01-15T18:48:13.974877+0000","last_fullsized":"2021-01-15T18:48:13.974877+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7743,"ondisk_log_size":7743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14712998,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":236177708,"num_objects":28845,"num_object_clones":0,"num_object_copies":115380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125880,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14627,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":473272320,"data_stored":473272320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125907,"ondisk_log_size":125907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365140,"kb_used_data":115824,"kb_used_omap":0,"kb_used_meta":249279,"kb_avail":94006700,"statfs":{"total":96636764160,"available":96262860800,"internally_reserved":0,"allocated":118603776,"data_stored":118350598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255261748},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,5,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":9.1289999999999996},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.505}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.89900000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.3030000000000008},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.995}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357752,"kb_used_data":116916,"kb_used_omap":0,"kb_used_meta":240768,"kb_avail":94014088,"statfs":{"total":96636764160,"available":96270426112,"internally_reserved":0,"allocated":119721984,"data_stored":119468806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":246546432},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,0,2,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":3.367}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":10.452999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":1.0369999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353104,"kb_used_data":116184,"kb_used_omap":0,"kb_used_meta":236863,"kb_avail":94018736,"statfs":{"total":96636764160,"available":96275185664,"internally_reserved":0,"allocated":118972416,"data_stored":118719238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":242547764},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":1.159},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.248}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":9.4139999999999997},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.5880000000000001},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":3.2559999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355120,"kb_used_data":115860,"kb_used_omap":0,"kb_used_meta":239231,"kb_avail":94016720,"statfs":{"total":96636764160,"available":96273121280,"internally_reserved":0,"allocated":118640640,"data_stored":118387462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244972596},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.351},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":6.9279999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.64600000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":6.734},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":6.8310000000000004}]}]}]}} 2021-01-15T18:48:20.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:21.997 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:22.013 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:22.083 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2021-01-15T18:48:20.621155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230983980,"num_objects":28211,"num_object_clones":0,"num_object_copies":112844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126514,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14671,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126541,"ondisk_log_size":126541,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1428544,"kb_used_data":452540,"kb_used_omap":2,"kb_used_meta":975869,"kb_avail":376058816,"statfs":{"total":386547056640,"available":385084227584,"internally_reserved":0,"allocated":463400960,"data_stored":462388248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":999290012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,26,17,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":99,"apply_latency_ms":99,"commit_latency_ns":99000000,"apply_latency_ns":99000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22528000,"num_objects":-2750,"num_object_clones":0,"num_object_copies":-11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013492"},"pg_stats":[{"pgid":"1.f","version":"48'7736","reported_seq":"8002","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.447074+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:20.447074+0000","last_peered":"2021-01-15T18:48:20.447074+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:20.447074+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7736,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14516224,"num_objects":1772,"num_object_clones":0,"num_object_copies":7088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7719","reported_seq":"10650","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.612091+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:15.612091+0000","last_peered":"2021-01-15T18:48:15.612091+0000","last_clean":"2021-01-15T18:48:15.612091+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:15.612091+0000","last_undegraded":"2021-01-15T18:48:15.612091+0000","last_fullsized":"2021-01-15T18:48:15.612091+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7719,"ondisk_log_size":7719,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14270630,"num_objects":1743,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7797","reported_seq":"10736","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:18.838903+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:18.838903+0000","last_peered":"2021-01-15T18:48:18.838903+0000","last_clean":"2021-01-15T18:48:18.838903+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:18.838903+0000","last_undegraded":"2021-01-15T18:48:18.838903+0000","last_fullsized":"2021-01-15T18:48:18.838903+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7797,"ondisk_log_size":7797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13881451,"num_objects":1696,"num_object_clones":0,"num_object_copies":6784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7796,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'8051","reported_seq":"8304","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.555352+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:20.555352+0000","last_peered":"2021-01-15T18:48:20.555352+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:20.555352+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8051,"ondisk_log_size":8051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14262272,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'8024","reported_seq":"8623","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.143720+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:15.143720+0000","last_peered":"2021-01-15T18:48:15.143720+0000","last_clean":"2021-01-15T18:48:15.143720+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:15.143720+0000","last_undegraded":"2021-01-15T18:48:15.143720+0000","last_fullsized":"2021-01-15T18:48:15.143720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8024,"ondisk_log_size":8024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14778534,"num_objects":1805,"num_object_clones":0,"num_object_copies":7220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7768","reported_seq":"7987","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.579835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:15.579835+0000","last_peered":"2021-01-15T18:48:15.579835+0000","last_clean":"2021-01-15T18:48:15.579835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:15.579835+0000","last_undegraded":"2021-01-15T18:48:15.579835+0000","last_fullsized":"2021-01-15T18:48:15.579835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7768,"ondisk_log_size":7768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14401702,"num_objects":1759,"num_object_clones":0,"num_object_copies":7036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7922","reported_seq":"10665","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.585415+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:20.585415+0000","last_peered":"2021-01-15T18:48:20.585415+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:20.585415+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7922,"ondisk_log_size":7922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14770342,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'8024","reported_seq":"8188","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:19.016031+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:19.016031+0000","last_peered":"2021-01-15T18:48:19.016031+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:19.016031+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8024,"ondisk_log_size":8024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15011923,"num_objects":1833,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":420}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'7976","reported_seq":"10919","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.598942+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:15.598942+0000","last_peered":"2021-01-15T18:48:15.598942+0000","last_clean":"2021-01-15T18:48:15.598942+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:15.598942+0000","last_undegraded":"2021-01-15T18:48:15.598942+0000","last_fullsized":"2021-01-15T18:48:15.598942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7976,"ondisk_log_size":7976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14356563,"num_objects":1753,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'8004","reported_seq":"10953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.303439+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:20.303439+0000","last_peered":"2021-01-15T18:48:20.303439+0000","last_clean":"2021-01-15T18:48:20.303439+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:20.303439+0000","last_undegraded":"2021-01-15T18:48:20.303439+0000","last_fullsized":"2021-01-15T18:48:20.303439+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8004,"ondisk_log_size":8004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14364921,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8001,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7825","reported_seq":"8073","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.400940+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:20.400940+0000","last_peered":"2021-01-15T18:48:20.400940+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:20.400940+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7825,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14164134,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":438}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7778","reported_seq":"10615","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.344402+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:15.344402+0000","last_peered":"2021-01-15T18:48:15.344402+0000","last_clean":"2021-01-15T18:48:15.344402+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:15.344402+0000","last_undegraded":"2021-01-15T18:48:15.344402+0000","last_fullsized":"2021-01-15T18:48:15.344402+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7778,"ondisk_log_size":7778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14655654,"num_objects":1790,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7776,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7756","reported_seq":"10659","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:19.516191+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:19.516191+0000","last_peered":"2021-01-15T18:48:19.516191+0000","last_clean":"2021-01-15T18:48:19.516191+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:19.516191+0000","last_undegraded":"2021-01-15T18:48:19.516191+0000","last_fullsized":"2021-01-15T18:48:19.516191+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7756,"ondisk_log_size":7756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14082214,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8092","reported_seq":"8326","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:15.411332+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:15.411332+0000","last_peered":"2021-01-15T18:48:15.411332+0000","last_clean":"2021-01-15T18:48:15.411332+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:15.411332+0000","last_undegraded":"2021-01-15T18:48:15.411332+0000","last_fullsized":"2021-01-15T18:48:15.411332+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8092,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14696614,"num_objects":1795,"num_object_clones":0,"num_object_copies":7180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8261","reported_seq":"8534","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.553369+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:20.553369+0000","last_peered":"2021-01-15T18:48:20.553369+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:20.553369+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8261,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14590284,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7808","reported_seq":"10681","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:18.909177+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:18.909177+0000","last_peered":"2021-01-15T18:48:18.909177+0000","last_clean":"2021-01-15T18:48:18.909177+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:18.909177+0000","last_undegraded":"2021-01-15T18:48:18.909177+0000","last_fullsized":"2021-01-15T18:48:18.909177+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7808,"ondisk_log_size":7808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14180518,"num_objects":1732,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":230983980,"num_objects":28211,"num_object_clones":0,"num_object_copies":112844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126514,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14671,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":460640256,"data_stored":460640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126541,"ondisk_log_size":126541,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350596,"kb_used_data":111644,"kb_used_omap":0,"kb_used_meta":238911,"kb_avail":94021244,"statfs":{"total":96636764160,"available":96277753856,"internally_reserved":0,"allocated":114323456,"data_stored":114070278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244644916},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,17,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.66500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.90900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.1110000000000002},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.478}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363688,"kb_used_data":112924,"kb_used_omap":0,"kb_used_meta":250752,"kb_avail":94008152,"statfs":{"total":96636764160,"available":96264347648,"internally_reserved":0,"allocated":115634176,"data_stored":115380998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":256770048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.83699999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359140,"kb_used_data":112112,"kb_used_omap":0,"kb_used_meta":246975,"kb_avail":94012700,"statfs":{"total":96636764160,"available":96269004800,"internally_reserved":0,"allocated":114802688,"data_stored":114549510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":252902452},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":1.3129999999999999},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.492}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":1.5229999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.1379999999999999},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":1.7509999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355120,"kb_used_data":115860,"kb_used_omap":0,"kb_used_meta":239231,"kb_avail":94016720,"statfs":{"total":96636764160,"available":96273121280,"internally_reserved":0,"allocated":118640640,"data_stored":118387462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244972596},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.351},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":6.9279999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.64600000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":6.734},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":6.8310000000000004}]}]}]}} 2021-01-15T18:48:22.085 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:23.406 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:23.406 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6736/111252 objects degraded (6.055%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":48,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":27813,"data_bytes":227723564,"bytes_used":1452249088,"bytes_avail":385094807552,"bytes_total":386547056640,"degraded_objects":6736,"degraded_total":111252,"degraded_ratio":0.060547226117283283,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":209},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:23.527 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:23.529 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:25.035 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:25.042 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:25.076 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2021-01-15T18:48:24.622226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226093356,"num_objects":27614,"num_object_clones":0,"num_object_copies":110456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27614,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127111,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14671,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127138,"ondisk_log_size":127138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1403724,"kb_used_data":444228,"kb_used_omap":2,"kb_used_meta":959357,"kb_avail":376083636,"statfs":{"total":386547056640,"available":385109643264,"internally_reserved":0,"allocated":454889472,"data_stored":453876760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":982381724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,7,28,33,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":134,"apply_latency_ms":134,"commit_latency_ns":134000000,"apply_latency_ns":134000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19038208,"num_objects":-2324,"num_object_clones":0,"num_object_copies":-9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013690"},"pg_stats":[{"pgid":"1.f","version":"48'7736","reported_seq":"8002","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.447074+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:20.447074+0000","last_peered":"2021-01-15T18:48:20.447074+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:20.447074+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7736,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14516224,"num_objects":1772,"num_object_clones":0,"num_object_copies":7088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7781","reported_seq":"10713","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.649573+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:20.649573+0000","last_peered":"2021-01-15T18:48:20.649573+0000","last_clean":"2021-01-15T18:48:20.649573+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:20.649573+0000","last_undegraded":"2021-01-15T18:48:20.649573+0000","last_fullsized":"2021-01-15T18:48:20.649573+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7781,"ondisk_log_size":7781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13762726,"num_objects":1681,"num_object_clones":0,"num_object_copies":6724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7858","reported_seq":"10797","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:23.691382+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:23.691382+0000","last_peered":"2021-01-15T18:48:23.691382+0000","last_clean":"2021-01-15T18:48:23.691382+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:23.691382+0000","last_undegraded":"2021-01-15T18:48:23.691382+0000","last_fullsized":"2021-01-15T18:48:23.691382+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7858,"ondisk_log_size":7858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13381739,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7857,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'8051","reported_seq":"8304","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.555352+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:20.555352+0000","last_peered":"2021-01-15T18:48:20.555352+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:20.555352+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8051,"ondisk_log_size":8051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14262272,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'8091","reported_seq":"8691","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.364998+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:20.364998+0000","last_peered":"2021-01-15T18:48:20.364998+0000","last_clean":"2021-01-15T18:48:20.364998+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:20.364998+0000","last_undegraded":"2021-01-15T18:48:20.364998+0000","last_fullsized":"2021-01-15T18:48:20.364998+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8091,"ondisk_log_size":8091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14229670,"num_objects":1738,"num_object_clones":0,"num_object_copies":6952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8089,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7842","reported_seq":"8061","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.633543+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:20.633543+0000","last_peered":"2021-01-15T18:48:20.633543+0000","last_clean":"2021-01-15T18:48:20.633543+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:20.633543+0000","last_undegraded":"2021-01-15T18:48:20.633543+0000","last_fullsized":"2021-01-15T18:48:20.633543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7842,"ondisk_log_size":7842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13795494,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7922","reported_seq":"10665","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.585415+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:20.585415+0000","last_peered":"2021-01-15T18:48:20.585415+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:20.585415+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7922,"ondisk_log_size":7922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14770342,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1340,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1748}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'8095","reported_seq":"8257","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:24.099961+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:24.099961+0000","last_peered":"2021-01-15T18:48:24.099961+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:24.099961+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8095,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14430291,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'8042","reported_seq":"10985","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.522438+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:20.522438+0000","last_peered":"2021-01-15T18:48:20.522438+0000","last_clean":"2021-01-15T18:48:20.522438+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:20.522438+0000","last_undegraded":"2021-01-15T18:48:20.522438+0000","last_fullsized":"2021-01-15T18:48:20.522438+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8042,"ondisk_log_size":8042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13815891,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'8004","reported_seq":"10953","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.303439+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:20.303439+0000","last_peered":"2021-01-15T18:48:20.303439+0000","last_clean":"2021-01-15T18:48:20.303439+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:20.303439+0000","last_undegraded":"2021-01-15T18:48:20.303439+0000","last_fullsized":"2021-01-15T18:48:20.303439+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8004,"ondisk_log_size":8004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14364921,"num_objects":1755,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8001,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7825","reported_seq":"8073","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.400940+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:20.400940+0000","last_peered":"2021-01-15T18:48:20.400940+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:20.400940+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7825,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14164134,"num_objects":1730,"num_object_clones":0,"num_object_copies":6920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":438}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7833","reported_seq":"10670","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.638383+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:20.638383+0000","last_peered":"2021-01-15T18:48:20.638383+0000","last_clean":"2021-01-15T18:48:20.638383+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:20.638383+0000","last_undegraded":"2021-01-15T18:48:20.638383+0000","last_fullsized":"2021-01-15T18:48:20.638383+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7833,"ondisk_log_size":7833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14205094,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7756","reported_seq":"10659","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:19.516191+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:19.516191+0000","last_peered":"2021-01-15T18:48:19.516191+0000","last_clean":"2021-01-15T18:48:19.516191+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:19.516191+0000","last_undegraded":"2021-01-15T18:48:19.516191+0000","last_fullsized":"2021-01-15T18:48:19.516191+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7756,"ondisk_log_size":7756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14082214,"num_objects":1720,"num_object_clones":0,"num_object_copies":6880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8166","reported_seq":"8400","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:20.523526+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:20.523526+0000","last_peered":"2021-01-15T18:48:20.523526+0000","last_clean":"2021-01-15T18:48:20.523526+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:20.523526+0000","last_undegraded":"2021-01-15T18:48:20.523526+0000","last_fullsized":"2021-01-15T18:48:20.523526+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8166,"ondisk_log_size":8166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14090406,"num_objects":1721,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8164,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8261","reported_seq":"8534","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:20.553369+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:20.553369+0000","last_peered":"2021-01-15T18:48:20.553369+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:20.553369+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8261,"ondisk_log_size":8261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14590284,"num_objects":1783,"num_object_clones":0,"num_object_copies":7132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7875","reported_seq":"10748","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:24.069382+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:24.069382+0000","last_peered":"2021-01-15T18:48:24.069382+0000","last_clean":"2021-01-15T18:48:24.069382+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:24.069382+0000","last_undegraded":"2021-01-15T18:48:24.069382+0000","last_fullsized":"2021-01-15T18:48:24.069382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7875,"ondisk_log_size":7875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631654,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":226093356,"num_objects":27614,"num_object_clones":0,"num_object_copies":110456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27614,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127111,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14671,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":452165632,"data_stored":452165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127138,"ondisk_log_size":127138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215210,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350596,"kb_used_data":111644,"kb_used_omap":0,"kb_used_meta":238911,"kb_avail":94021244,"statfs":{"total":96636764160,"available":96277753856,"internally_reserved":0,"allocated":114323456,"data_stored":114070278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244644916},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,17,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.66500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.90900000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.1110000000000002},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.478}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349200,"kb_used_data":108744,"kb_used_omap":0,"kb_used_meta":240448,"kb_avail":94022640,"statfs":{"total":96636764160,"available":96279183360,"internally_reserved":0,"allocated":111353856,"data_stored":111100678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":246218752},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.6630000000000003},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":7.2249999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":6.9880000000000004},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":6.7050000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":10.246},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":10.759}]}]},{"osd":0,"up_from":22,"seq":94489280622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359140,"kb_used_data":112112,"kb_used_omap":0,"kb_used_meta":246975,"kb_avail":94012700,"statfs":{"total":96636764160,"available":96269004800,"internally_reserved":0,"allocated":114802688,"data_stored":114549510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":252902452},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":1.3129999999999999},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.492}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":1.5229999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.1379999999999999},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":1.7509999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344788,"kb_used_data":111728,"kb_used_omap":0,"kb_used_meta":233023,"kb_avail":94027052,"statfs":{"total":96636764160,"available":96283701248,"internally_reserved":0,"allocated":114409472,"data_stored":114156294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":238615604},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.64600000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.052}]}]}]}} 2021-01-15T18:48:28.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:28.715 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:28.720 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:28.742 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2021-01-15T18:48:28.623106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":219318572,"num_objects":26787,"num_object_clones":0,"num_object_copies":107148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127938,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14716,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127965,"ondisk_log_size":127965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1417632,"kb_used_data":431968,"kb_used_omap":2,"kb_used_meta":985469,"kb_avail":376069728,"statfs":{"total":386547056640,"available":385095401472,"internally_reserved":0,"allocated":442335232,"data_stored":441322520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1009120412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,16,3,10,25,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11665408,"num_objects":-1424,"num_object_clones":0,"num_object_copies":-5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-45,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1424,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001951"},"pg_stats":[{"pgid":"1.f","version":"48'7802","reported_seq":"8068","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.377211+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:25.377211+0000","last_peered":"2021-01-15T18:48:25.377211+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:25.377211+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7802,"ondisk_log_size":7802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13975552,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7802,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7848","reported_seq":"10780","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.653024+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:25.653024+0000","last_peered":"2021-01-15T18:48:25.653024+0000","last_clean":"2021-01-15T18:48:25.653024+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:25.653024+0000","last_undegraded":"2021-01-15T18:48:25.653024+0000","last_fullsized":"2021-01-15T18:48:25.653024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7848,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13213862,"num_objects":1614,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7858","reported_seq":"10797","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:23.691382+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:23.691382+0000","last_peered":"2021-01-15T18:48:23.691382+0000","last_clean":"2021-01-15T18:48:23.691382+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:23.691382+0000","last_undegraded":"2021-01-15T18:48:23.691382+0000","last_fullsized":"2021-01-15T18:48:23.691382+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7858,"ondisk_log_size":7858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13381739,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7857,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'8112","reported_seq":"8366","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.391717+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:25.391717+0000","last_peered":"2021-01-15T18:48:25.391717+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:25.391717+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8112,"ondisk_log_size":8112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13762560,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'8156","reported_seq":"8755","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.712029+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:25.712029+0000","last_peered":"2021-01-15T18:48:25.712029+0000","last_clean":"2021-01-15T18:48:25.712029+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:25.712029+0000","last_undegraded":"2021-01-15T18:48:25.712029+0000","last_fullsized":"2021-01-15T18:48:25.712029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8156,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13697190,"num_objects":1673,"num_object_clones":0,"num_object_copies":6692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7905","reported_seq":"8124","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.600580+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:25.600580+0000","last_peered":"2021-01-15T18:48:25.600580+0000","last_clean":"2021-01-15T18:48:25.600580+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:25.600580+0000","last_undegraded":"2021-01-15T18:48:25.600580+0000","last_fullsized":"2021-01-15T18:48:25.600580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7905,"ondisk_log_size":7905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13279398,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7993","reported_seq":"10826","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.560838+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:25.560838+0000","last_peered":"2021-01-15T18:48:25.560838+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:25.560838+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7993,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14188710,"num_objects":1733,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'8095","reported_seq":"8257","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:24.099961+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:24.099961+0000","last_peered":"2021-01-15T18:48:24.099961+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:24.099961+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8095,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14430291,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'8083","reported_seq":"11026","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.744980+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:25.744980+0000","last_peered":"2021-01-15T18:48:25.744980+0000","last_clean":"2021-01-15T18:48:25.744980+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:25.744980+0000","last_undegraded":"2021-01-15T18:48:25.744980+0000","last_fullsized":"2021-01-15T18:48:25.744980+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8083,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13480019,"num_objects":1646,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'8077","reported_seq":"11026","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.517837+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:25.517837+0000","last_peered":"2021-01-15T18:48:25.517837+0000","last_clean":"2021-01-15T18:48:25.517837+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:25.517837+0000","last_undegraded":"2021-01-15T18:48:25.517837+0000","last_fullsized":"2021-01-15T18:48:25.517837+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8077,"ondisk_log_size":8077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13766905,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7890","reported_seq":"8138","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.561176+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:25.561176+0000","last_peered":"2021-01-15T18:48:25.561176+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:25.561176+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7890,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631654,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7905","reported_seq":"10741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.731209+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:25.731209+0000","last_peered":"2021-01-15T18:48:25.731209+0000","last_clean":"2021-01-15T18:48:25.731209+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:25.731209+0000","last_undegraded":"2021-01-15T18:48:25.731209+0000","last_fullsized":"2021-01-15T18:48:25.731209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7905,"ondisk_log_size":7905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13615270,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7814","reported_seq":"10717","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:24.563623+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:24.563623+0000","last_peered":"2021-01-15T18:48:24.563623+0000","last_clean":"2021-01-15T18:48:24.563623+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:24.563623+0000","last_undegraded":"2021-01-15T18:48:24.563623+0000","last_fullsized":"2021-01-15T18:48:24.563623+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7814,"ondisk_log_size":7814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13607078,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8232","reported_seq":"8466","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.616826+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:25.616826+0000","last_peered":"2021-01-15T18:48:25.616826+0000","last_clean":"2021-01-15T18:48:25.616826+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:25.616826+0000","last_undegraded":"2021-01-15T18:48:25.616826+0000","last_fullsized":"2021-01-15T18:48:25.616826+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8232,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13549734,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8320","reported_seq":"8594","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.533957+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:25.533957+0000","last_peered":"2021-01-15T18:48:25.533957+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:25.533957+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8320,"ondisk_log_size":8320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14106956,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7875","reported_seq":"10748","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:24.069382+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:24.069382+0000","last_peered":"2021-01-15T18:48:24.069382+0000","last_clean":"2021-01-15T18:48:24.069382+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:24.069382+0000","last_undegraded":"2021-01-15T18:48:24.069382+0000","last_fullsized":"2021-01-15T18:48:24.069382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7875,"ondisk_log_size":7875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631654,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":219318572,"num_objects":26787,"num_object_clones":0,"num_object_copies":107148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127938,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14716,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":439717888,"data_stored":439717888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127965,"ondisk_log_size":127965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356512,"kb_used_data":107524,"kb_used_omap":0,"kb_used_meta":248959,"kb_avail":94015328,"statfs":{"total":96636764160,"available":96271695872,"internally_reserved":0,"allocated":110104576,"data_stored":109851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":254934068},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,2,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.77100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.1899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":16.936}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349200,"kb_used_data":108744,"kb_used_omap":0,"kb_used_meta":240448,"kb_avail":94022640,"statfs":{"total":96636764160,"available":96279183360,"internally_reserved":0,"allocated":111353856,"data_stored":111100678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":246218752},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.6630000000000003},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":7.2249999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":6.9880000000000004},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":6.7050000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":10.246},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":10.759}]}]},{"osd":0,"up_from":22,"seq":94489280623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365144,"kb_used_data":108124,"kb_used_omap":0,"kb_used_meta":256959,"kb_avail":94006696,"statfs":{"total":96636764160,"available":96262856704,"internally_reserved":0,"allocated":110718976,"data_stored":110465798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263126068},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,11,0,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.628},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":8.9100000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.2400000000000002},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":3.641}]}]},{"osd":1,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346776,"kb_used_data":107576,"kb_used_omap":0,"kb_used_meta":239103,"kb_avail":94025064,"statfs":{"total":96636764160,"available":96281665536,"internally_reserved":0,"allocated":110157824,"data_stored":109904646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244841524},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":4.819},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":2.6800000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":6.1120000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":6.2160000000000002},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":6.3070000000000004}]}]}]}} 2021-01-15T18:48:28.743 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:29.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:29.324 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6691/107148 objects degraded (6.245%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":559,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26787,"data_bytes":219318572,"bytes_used":1451655168,"bytes_avail":385095401472,"bytes_total":386547056640,"degraded_objects":6691,"degraded_total":107148,"degraded_ratio":0.062446335909209692,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:29.337 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:29.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:29.906 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:29.907 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:29.919 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2021-01-15T18:48:28.623106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":219318572,"num_objects":26787,"num_object_clones":0,"num_object_copies":107148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127938,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14716,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127965,"ondisk_log_size":127965,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1417632,"kb_used_data":431968,"kb_used_omap":2,"kb_used_meta":985469,"kb_avail":376069728,"statfs":{"total":386547056640,"available":385095401472,"internally_reserved":0,"allocated":442335232,"data_stored":441322520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1009120412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,16,3,10,25,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11665408,"num_objects":-1424,"num_object_clones":0,"num_object_copies":-5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-45,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1424,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001951"},"pg_stats":[{"pgid":"1.f","version":"48'7802","reported_seq":"8068","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.377211+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:25.377211+0000","last_peered":"2021-01-15T18:48:25.377211+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:25.377211+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7802,"ondisk_log_size":7802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13975552,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7802,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"48'7848","reported_seq":"10780","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.653024+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:25.653024+0000","last_peered":"2021-01-15T18:48:25.653024+0000","last_clean":"2021-01-15T18:48:25.653024+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:25.653024+0000","last_undegraded":"2021-01-15T18:48:25.653024+0000","last_fullsized":"2021-01-15T18:48:25.653024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7848,"ondisk_log_size":7848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13213862,"num_objects":1614,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"48'7858","reported_seq":"10797","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:23.691382+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:23.691382+0000","last_peered":"2021-01-15T18:48:23.691382+0000","last_clean":"2021-01-15T18:48:23.691382+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:23.691382+0000","last_undegraded":"2021-01-15T18:48:23.691382+0000","last_fullsized":"2021-01-15T18:48:23.691382+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7858,"ondisk_log_size":7858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13381739,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7857,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"48'8112","reported_seq":"8366","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.391717+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:25.391717+0000","last_peered":"2021-01-15T18:48:25.391717+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:25.391717+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8112,"ondisk_log_size":8112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13762560,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"48'8156","reported_seq":"8755","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.712029+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:25.712029+0000","last_peered":"2021-01-15T18:48:25.712029+0000","last_clean":"2021-01-15T18:48:25.712029+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:25.712029+0000","last_undegraded":"2021-01-15T18:48:25.712029+0000","last_fullsized":"2021-01-15T18:48:25.712029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8156,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13697190,"num_objects":1673,"num_object_clones":0,"num_object_copies":6692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"48'7905","reported_seq":"8124","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.600580+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:25.600580+0000","last_peered":"2021-01-15T18:48:25.600580+0000","last_clean":"2021-01-15T18:48:25.600580+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:25.600580+0000","last_undegraded":"2021-01-15T18:48:25.600580+0000","last_fullsized":"2021-01-15T18:48:25.600580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7905,"ondisk_log_size":7905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13279398,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'7993","reported_seq":"10826","reported_epoch":"48","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.560838+0000","last_change":"2021-01-15T18:46:28.790938+0000","last_active":"2021-01-15T18:48:25.560838+0000","last_peered":"2021-01-15T18:48:25.560838+0000","last_clean":"2021-01-15T18:39:33.836275+0000","last_became_active":"2021-01-15T18:40:32.747091+0000","last_became_peered":"2021-01-15T18:40:32.747091+0000","last_unstale":"2021-01-15T18:48:25.560838+0000","last_undegraded":"2021-01-15T18:40:32.512403+0000","last_fullsized":"2021-01-15T18:40:32.481871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7993,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14188710,"num_objects":1733,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1722}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"48'8095","reported_seq":"8257","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:24.099961+0000","last_change":"2021-01-15T18:40:32.778522+0000","last_active":"2021-01-15T18:48:24.099961+0000","last_peered":"2021-01-15T18:48:24.099961+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:24.099961+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8095,"ondisk_log_size":8095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14430291,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":349}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"48'8083","reported_seq":"11026","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.744980+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:25.744980+0000","last_peered":"2021-01-15T18:48:25.744980+0000","last_clean":"2021-01-15T18:48:25.744980+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:25.744980+0000","last_undegraded":"2021-01-15T18:48:25.744980+0000","last_fullsized":"2021-01-15T18:48:25.744980+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8083,"ondisk_log_size":8083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13480019,"num_objects":1646,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"48'8077","reported_seq":"11026","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.517837+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:25.517837+0000","last_peered":"2021-01-15T18:48:25.517837+0000","last_clean":"2021-01-15T18:48:25.517837+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:25.517837+0000","last_undegraded":"2021-01-15T18:48:25.517837+0000","last_fullsized":"2021-01-15T18:48:25.517837+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8077,"ondisk_log_size":8077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13766905,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"48'7890","reported_seq":"8138","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.561176+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:25.561176+0000","last_peered":"2021-01-15T18:48:25.561176+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:25.561176+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7890,"ondisk_log_size":7890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631654,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":373}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"48'7905","reported_seq":"10741","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.731209+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:25.731209+0000","last_peered":"2021-01-15T18:48:25.731209+0000","last_clean":"2021-01-15T18:48:25.731209+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:25.731209+0000","last_undegraded":"2021-01-15T18:48:25.731209+0000","last_fullsized":"2021-01-15T18:48:25.731209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7905,"ondisk_log_size":7905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13615270,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"48'7814","reported_seq":"10717","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:24.563623+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:24.563623+0000","last_peered":"2021-01-15T18:48:24.563623+0000","last_clean":"2021-01-15T18:48:24.563623+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:24.563623+0000","last_undegraded":"2021-01-15T18:48:24.563623+0000","last_fullsized":"2021-01-15T18:48:24.563623+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7814,"ondisk_log_size":7814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13607078,"num_objects":1662,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"48'8232","reported_seq":"8466","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:25.616826+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:25.616826+0000","last_peered":"2021-01-15T18:48:25.616826+0000","last_clean":"2021-01-15T18:48:25.616826+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:25.616826+0000","last_undegraded":"2021-01-15T18:48:25.616826+0000","last_fullsized":"2021-01-15T18:48:25.616826+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8232,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13549734,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8230,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"48'8320","reported_seq":"8594","reported_epoch":"48","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:25.533957+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:25.533957+0000","last_peered":"2021-01-15T18:48:25.533957+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:25.533957+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8320,"ondisk_log_size":8320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14106956,"num_objects":1724,"num_object_clones":0,"num_object_copies":6896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"48'7875","reported_seq":"10748","reported_epoch":"48","state":"active+clean","last_fresh":"2021-01-15T18:48:24.069382+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:24.069382+0000","last_peered":"2021-01-15T18:48:24.069382+0000","last_clean":"2021-01-15T18:48:24.069382+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:24.069382+0000","last_undegraded":"2021-01-15T18:48:24.069382+0000","last_fullsized":"2021-01-15T18:48:24.069382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7875,"ondisk_log_size":7875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631654,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":219318572,"num_objects":26787,"num_object_clones":0,"num_object_copies":107148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127938,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14716,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":439717888,"data_stored":439717888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127965,"ondisk_log_size":127965,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215211,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356512,"kb_used_data":107524,"kb_used_omap":0,"kb_used_meta":248959,"kb_avail":94015328,"statfs":{"total":96636764160,"available":96271695872,"internally_reserved":0,"allocated":110104576,"data_stored":109851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":254934068},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,2,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.77100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.1899999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":16.936}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349200,"kb_used_data":108744,"kb_used_omap":0,"kb_used_meta":240448,"kb_avail":94022640,"statfs":{"total":96636764160,"available":96279183360,"internally_reserved":0,"allocated":111353856,"data_stored":111100678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":246218752},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.6630000000000003},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":7.2249999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":6.9880000000000004},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":6.7050000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":10.246},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":10.759}]}]},{"osd":0,"up_from":22,"seq":94489280623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365144,"kb_used_data":108124,"kb_used_omap":0,"kb_used_meta":256959,"kb_avail":94006696,"statfs":{"total":96636764160,"available":96262856704,"internally_reserved":0,"allocated":110718976,"data_stored":110465798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263126068},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,11,0,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.628},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71999999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":8.9100000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.2400000000000002},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":3.641}]}]},{"osd":1,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346776,"kb_used_data":107576,"kb_used_omap":0,"kb_used_meta":239103,"kb_avail":94025064,"statfs":{"total":96636764160,"available":96281665536,"internally_reserved":0,"allocated":110157824,"data_stored":109904646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":244841524},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":4.819},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":2.6800000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":6.1120000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":6.2160000000000002},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":6.3070000000000004}]}]}]}} 2021-01-15T18:48:32.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:33.226 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:33.227 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:33.239 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2021-01-15T18:48:32.623888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214436140,"num_objects":26191,"num_object_clones":0,"num_object_copies":104764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14783,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128561,"ondisk_log_size":128561,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1416156,"kb_used_data":420404,"kb_used_omap":2,"kb_used_meta":995709,"kb_avail":376071204,"statfs":{"total":386547056640,"available":385096912896,"internally_reserved":0,"allocated":430493696,"data_stored":429441244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1019606172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,3,1,8,3,1,0,2,5,10],"upper_bound":4096},"perf_stat":{"commit_latency_ms":91,"apply_latency_ms":91,"commit_latency_ns":91000000,"apply_latency_ns":91000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11657216,"num_objects":-1423,"num_object_clones":0,"num_object_copies":-5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001661"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7907","reported_seq":"10848","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.587469+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:28.587469+0000","last_peered":"2021-01-15T18:48:28.587469+0000","last_clean":"2021-01-15T18:48:28.587469+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:28.587469+0000","last_undegraded":"2021-01-15T18:48:28.587469+0000","last_fullsized":"2021-01-15T18:48:28.587469+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7907,"ondisk_log_size":7907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12980331,"num_objects":1586,"num_object_clones":0,"num_object_copies":6344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7906,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8145","reported_seq":"8424","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.117178+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:29.117178+0000","last_peered":"2021-01-15T18:48:29.117178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:29.117178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8145,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14020691,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8144,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214436140,"num_objects":26191,"num_object_clones":0,"num_object_copies":104764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14783,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428691456,"data_stored":428691456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128561,"ondisk_log_size":128561,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.67200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.61299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355604,"kb_used_data":105220,"kb_used_omap":0,"kb_used_meta":250368,"kb_avail":94016236,"statfs":{"total":96636764160,"available":96272625664,"internally_reserved":0,"allocated":107745280,"data_stored":107482167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":256376832},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":2.8380000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":3.2149999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.59999999999999998},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.573},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":2.0059999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341632,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":236543,"kb_avail":94030208,"statfs":{"total":96636764160,"available":96286932992,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":242220084},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,2,0,0,2,5,10],"upper_bound":4096},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.839},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":2.0430000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.274},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.78000000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.871},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.95399999999999996}]}]}]}} 2021-01-15T18:48:33.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:33.627 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6624/104764 objects degraded (6.323%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":564,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26191,"data_bytes":214436140,"bytes_used":1450143744,"bytes_avail":385096912896,"bytes_total":386547056640,"degraded_objects":6624,"degraded_total":104764,"degraded_ratio":0.063227826352563854,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:33.641 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:33.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:33.951 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:33.952 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:33.963 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2021-01-15T18:48:32.623888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214436140,"num_objects":26191,"num_object_clones":0,"num_object_copies":104764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14783,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128561,"ondisk_log_size":128561,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1416156,"kb_used_data":420404,"kb_used_omap":2,"kb_used_meta":995709,"kb_avail":376071204,"statfs":{"total":386547056640,"available":385096912896,"internally_reserved":0,"allocated":430493696,"data_stored":429441244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1019606172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,3,1,8,3,1,0,2,5,10],"upper_bound":4096},"perf_stat":{"commit_latency_ms":91,"apply_latency_ms":91,"commit_latency_ns":91000000,"apply_latency_ns":91000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11657216,"num_objects":-1423,"num_object_clones":0,"num_object_copies":-5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001661"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7907","reported_seq":"10848","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.587469+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:28.587469+0000","last_peered":"2021-01-15T18:48:28.587469+0000","last_clean":"2021-01-15T18:48:28.587469+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:28.587469+0000","last_undegraded":"2021-01-15T18:48:28.587469+0000","last_fullsized":"2021-01-15T18:48:28.587469+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7907,"ondisk_log_size":7907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12980331,"num_objects":1586,"num_object_clones":0,"num_object_copies":6344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7906,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8145","reported_seq":"8424","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.117178+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:29.117178+0000","last_peered":"2021-01-15T18:48:29.117178+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:29.117178+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8145,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14020691,"num_objects":1712,"num_object_clones":0,"num_object_copies":6848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8144,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214436140,"num_objects":26191,"num_object_clones":0,"num_object_copies":104764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128534,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14783,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428691456,"data_stored":428691456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128561,"ondisk_log_size":128561,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.67200000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.61299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355604,"kb_used_data":105220,"kb_used_omap":0,"kb_used_meta":250368,"kb_avail":94016236,"statfs":{"total":96636764160,"available":96272625664,"internally_reserved":0,"allocated":107745280,"data_stored":107482167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":256376832},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":2.8380000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":3.2149999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.59999999999999998},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":2.573},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":2.0059999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341632,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":236543,"kb_avail":94030208,"statfs":{"total":96636764160,"available":96286932992,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":242220084},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,2,0,0,2,5,10],"upper_bound":4096},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.839},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":2.0430000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":1.274},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.78000000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.871},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.95399999999999996}]}]}]}} 2021-01-15T18:48:36.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:37.275 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:37.276 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:37.287 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2021-01-15T18:48:36.624640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15282,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1433132,"kb_used_data":420236,"kb_used_omap":2,"kb_used_meta":1012861,"kb_avail":376054228,"statfs":{"total":386547056640,"available":385079529472,"internally_reserved":0,"allocated":430321664,"data_stored":429269212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1037169820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4907008,"num_objects":-599,"num_object_clones":0,"num_object_copies":-2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.711919"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7909","reported_seq":"10850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.207166+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:29.207166+0000","last_peered":"2021-01-15T18:48:29.207166+0000","last_clean":"2021-01-15T18:48:29.207166+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:29.207166+0000","last_undegraded":"2021-01-15T18:48:29.207166+0000","last_fullsized":"2021-01-15T18:48:29.207166+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7909,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12963947,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7908,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8146","reported_seq":"9423","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:34.137554+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:34.137554+0000","last_peered":"2021-01-15T18:48:34.137554+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:34.137554+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8146,"ondisk_log_size":8146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14012499,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":852}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15282,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428650496,"data_stored":428650496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.93100000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.67200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359588,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":254528,"kb_avail":94012252,"statfs":{"total":96636764160,"available":96268546048,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":260636672},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.58599999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.629},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.73699999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.76300000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354624,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":249535,"kb_avail":94017216,"statfs":{"total":96636764160,"available":96273629184,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255523892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.41499999999999998},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.78800000000000003}]}]}]}} 2021-01-15T18:48:37.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:37.638 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:37.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6125/104752 objects degraded (5.847%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":568,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26188,"data_bytes":214411564,"bytes_used":1467527168,"bytes_avail":385079529472,"bytes_total":386547056640,"degraded_objects":6125,"degraded_total":104752,"degraded_ratio":0.058471437299526502,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":61},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:37.653 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:37.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:37.949 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:37.951 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:37.963 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2021-01-15T18:48:36.624640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15282,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1433132,"kb_used_data":420236,"kb_used_omap":2,"kb_used_meta":1012861,"kb_avail":376054228,"statfs":{"total":386547056640,"available":385079529472,"internally_reserved":0,"allocated":430321664,"data_stored":429269212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1037169820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4907008,"num_objects":-599,"num_object_clones":0,"num_object_copies":-2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.711919"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7909","reported_seq":"10850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.207166+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:29.207166+0000","last_peered":"2021-01-15T18:48:29.207166+0000","last_clean":"2021-01-15T18:48:29.207166+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:29.207166+0000","last_undegraded":"2021-01-15T18:48:29.207166+0000","last_fullsized":"2021-01-15T18:48:29.207166+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7909,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12963947,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7908,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8146","reported_seq":"9423","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:34.137554+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:34.137554+0000","last_peered":"2021-01-15T18:48:34.137554+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:34.137554+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8146,"ondisk_log_size":8146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14012499,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":852}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15282,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428650496,"data_stored":428650496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.93100000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.67200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359588,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":254528,"kb_avail":94012252,"statfs":{"total":96636764160,"available":96268546048,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":260636672},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:47:35 2021","interfaces":[{"interface":"back","average":{"1min":4.8099999999999996,"5min":3.2610000000000001,"15min":2.633},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.40799999999999997},"max":{"1min":16.481000000000002,"5min":20.885000000000002,"15min":20.885000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.077,"5min":2.0409999999999999,"15min":2.028},"min":{"1min":0.58399999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":16.587,"5min":20.98,"15min":20.98},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.58599999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:47:36 2021","interfaces":[{"interface":"back","average":{"1min":11.029,"5min":5.4749999999999996,"15min":3.681},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":50.948,"5min":50.948,"15min":50.948},"last":0.629},{"interface":"front","average":{"1min":8.4339999999999993,"5min":5.319,"15min":3.2850000000000001},"min":{"1min":0.38800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":50.031999999999996,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.73699999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.76300000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354624,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":249535,"kb_avail":94017216,"statfs":{"total":96636764160,"available":96273629184,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255523892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.41499999999999998},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.78800000000000003}]}]}]}} 2021-01-15T18:48:40.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:41.640 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:41.644 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:41.706 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2021-01-15T18:48:40.625384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15831,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1433132,"kb_used_data":420236,"kb_used_omap":2,"kb_used_meta":1012861,"kb_avail":376054228,"statfs":{"total":386547056640,"available":385079529472,"internally_reserved":0,"allocated":430321664,"data_stored":429269212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1037169820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4907008,"num_objects":-599,"num_object_clones":0,"num_object_copies":-2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002277"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7909","reported_seq":"10850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.207166+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:29.207166+0000","last_peered":"2021-01-15T18:48:29.207166+0000","last_clean":"2021-01-15T18:48:29.207166+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:29.207166+0000","last_undegraded":"2021-01-15T18:48:29.207166+0000","last_fullsized":"2021-01-15T18:48:29.207166+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7909,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12963947,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7908,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8146","reported_seq":"10521","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:39.143074+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:39.143074+0000","last_peered":"2021-01-15T18:48:39.143074+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:39.143074+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8146,"ondisk_log_size":8146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14012499,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1401}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15831,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428650496,"data_stored":428650496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215213,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":0.93100000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.67200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359588,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":254528,"kb_avail":94012252,"statfs":{"total":96636764160,"available":96268546048,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":260636672},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.70399999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.46100000000000002},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.443},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.78100000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":354624,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":249535,"kb_avail":94017216,"statfs":{"total":96636764160,"available":96273629184,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255523892},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,10,10],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.41499999999999998},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.78800000000000003}]}]}]}} 2021-01-15T18:48:41.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:43.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:43.252 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5576/104752 objects degraded (5.323%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26188,"data_bytes":214411564,"bytes_used":1459859456,"bytes_avail":385087197184,"bytes_total":386547056640,"degraded_objects":5576,"degraded_total":104752,"degraded_ratio":0.053230487246066903,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":32},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2021-01-15T18:48:12.608610+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:43.320 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:43.321 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:44.568 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:44.602 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:44.711 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2021-01-15T18:48:42.625768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15831,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1425644,"kb_used_data":420236,"kb_used_omap":2,"kb_used_meta":1005373,"kb_avail":376061716,"statfs":{"total":386547056640,"available":385087197184,"internally_reserved":0,"allocated":430321664,"data_stored":429269212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1029502108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3211264,"num_objects":-392,"num_object_clones":0,"num_object_copies":-1568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"1.f","version":"50'7844","reported_seq":"8112","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.262405+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:28.262405+0000","last_peered":"2021-01-15T18:48:28.262405+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:28.262405+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7844,"ondisk_log_size":7844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13631488,"num_objects":1664,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7882","reported_seq":"10816","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.956918+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:28.956918+0000","last_peered":"2021-01-15T18:48:28.956918+0000","last_clean":"2021-01-15T18:48:28.956918+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:28.956918+0000","last_undegraded":"2021-01-15T18:48:28.956918+0000","last_fullsized":"2021-01-15T18:48:28.956918+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7882,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12935334,"num_objects":1580,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7909","reported_seq":"10850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.207166+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:29.207166+0000","last_peered":"2021-01-15T18:48:29.207166+0000","last_clean":"2021-01-15T18:48:29.207166+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:29.207166+0000","last_undegraded":"2021-01-15T18:48:29.207166+0000","last_fullsized":"2021-01-15T18:48:29.207166+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7909,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12963947,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7908,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8148","reported_seq":"8404","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.192583+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:29.192583+0000","last_peered":"2021-01-15T18:48:29.192583+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:29.192583+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8148,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13467648,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":310}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8187","reported_seq":"8789","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.996252+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:28.996252+0000","last_peered":"2021-01-15T18:48:28.996252+0000","last_clean":"2021-01-15T18:48:28.996252+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:28.996252+0000","last_undegraded":"2021-01-15T18:48:28.996252+0000","last_fullsized":"2021-01-15T18:48:28.996252+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13443238,"num_objects":1642,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7935","reported_seq":"8156","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.841835+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:28.841835+0000","last_peered":"2021-01-15T18:48:28.841835+0000","last_clean":"2021-01-15T18:48:28.841835+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:28.841835+0000","last_undegraded":"2021-01-15T18:48:28.841835+0000","last_fullsized":"2021-01-15T18:48:28.841835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7935,"ondisk_log_size":7935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13033638,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"48'8008","reported_seq":"10878","reported_epoch":"50","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:28.198498+0000","last_change":"2021-01-15T18:48:28.198498+0000","last_active":"2021-01-15T18:48:28.198498+0000","last_peered":"2021-01-15T18:48:28.198498+0000","last_clean":"2021-01-15T18:48:28.198498+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:28.198498+0000","last_undegraded":"2021-01-15T18:48:28.198498+0000","last_fullsized":"2021-01-15T18:48:28.198498+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14065830,"num_objects":1718,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8146","reported_seq":"10521","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:39.143074+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:39.143074+0000","last_peered":"2021-01-15T18:48:39.143074+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:39.143074+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8146,"ondisk_log_size":8146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14012499,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1401}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8114","reported_seq":"11059","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.773938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:28.773938+0000","last_peered":"2021-01-15T18:48:28.773938+0000","last_clean":"2021-01-15T18:48:28.773938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:28.773938+0000","last_undegraded":"2021-01-15T18:48:28.773938+0000","last_fullsized":"2021-01-15T18:48:28.773938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8114,"ondisk_log_size":8114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13226067,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8115","reported_seq":"11066","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.945097+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:28.945097+0000","last_peered":"2021-01-15T18:48:28.945097+0000","last_clean":"2021-01-15T18:48:28.945097+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:28.945097+0000","last_undegraded":"2021-01-15T18:48:28.945097+0000","last_fullsized":"2021-01-15T18:48:28.945097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8115,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13455609,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7924","reported_seq":"8174","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:28.914106+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:28.914106+0000","last_peered":"2021-01-15T18:48:28.914106+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:28.914106+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7924,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13353126,"num_objects":1631,"num_object_clones":0,"num_object_copies":6524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'7932","reported_seq":"10771","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.045103+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:29.045103+0000","last_peered":"2021-01-15T18:48:29.045103+0000","last_clean":"2021-01-15T18:48:29.045103+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:29.045103+0000","last_undegraded":"2021-01-15T18:48:29.045103+0000","last_fullsized":"2021-01-15T18:48:29.045103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13394086,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7863","reported_seq":"10768","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.067688+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:29.067688+0000","last_peered":"2021-01-15T18:48:29.067688+0000","last_clean":"2021-01-15T18:48:29.067688+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:29.067688+0000","last_undegraded":"2021-01-15T18:48:29.067688+0000","last_fullsized":"2021-01-15T18:48:29.067688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7863,"ondisk_log_size":7863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13205670,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8264","reported_seq":"8500","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:28.336395+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:28.336395+0000","last_peered":"2021-01-15T18:48:28.336395+0000","last_clean":"2021-01-15T18:48:28.336395+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:28.336395+0000","last_undegraded":"2021-01-15T18:48:28.336395+0000","last_fullsized":"2021-01-15T18:48:28.336395+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8264,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13287590,"num_objects":1623,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8359","reported_seq":"8635","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:29.098537+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:29.098537+0000","last_peered":"2021-01-15T18:48:29.098537+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:29.098537+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8359,"ondisk_log_size":8359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13787468,"num_objects":1685,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7934","reported_seq":"10809","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:29.044401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:29.044401+0000","last_peered":"2021-01-15T18:48:29.044401+0000","last_clean":"2021-01-15T18:48:29.044401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:29.044401+0000","last_undegraded":"2021-01-15T18:48:29.044401+0000","last_fullsized":"2021-01-15T18:48:29.044401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7934,"ondisk_log_size":7934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13148326,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":214411564,"num_objects":26188,"num_object_clones":0,"num_object_copies":104752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128537,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15831,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428650496,"data_stored":428650496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128564,"ondisk_log_size":128564,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363492,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":258431,"kb_avail":94008348,"statfs":{"total":96636764160,"available":96264548352,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264633396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.75700000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:47:42 2021","interfaces":[{"interface":"back","average":{"1min":3.4710000000000001,"5min":4.3390000000000004,"15min":3.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.48599999999999999,"15min":0.42799999999999999},"max":{"1min":14.289,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":3.8610000000000002,"5min":4.2709999999999999,"15min":2.7789999999999999},"min":{"1min":0.78900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":16.010000000000002,"5min":26.635000000000002,"15min":26.635000000000002},"last":1.008}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.88900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359588,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":254528,"kb_avail":94012252,"statfs":{"total":96636764160,"available":96268546048,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":260636672},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.70399999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355428,"kb_used_data":105052,"kb_used_omap":0,"kb_used_meta":250367,"kb_avail":94016412,"statfs":{"total":96636764160,"available":96272805888,"internally_reserved":0,"allocated":107573248,"data_stored":107310135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":256375860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.46100000000000002},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.443},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.78100000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347136,"kb_used_data":105080,"kb_used_omap":0,"kb_used_meta":242047,"kb_avail":94024704,"statfs":{"total":96636764160,"available":96281296896,"internally_reserved":0,"allocated":107601920,"data_stored":107338807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":247856180},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,20],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":0.46000000000000002},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":0.51700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.76900000000000002}]}]}]}} 2021-01-15T18:48:47.713 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:49.214 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:49.233 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2021-01-15T18:48:48.731281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":206932268,"num_objects":25275,"num_object_clones":0,"num_object_copies":101100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129450,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16091,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129477,"ondisk_log_size":129477,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1411000,"kb_used_data":408028,"kb_used_omap":2,"kb_used_meta":1002877,"kb_avail":376076360,"statfs":{"total":386547056640,"available":385102192640,"internally_reserved":0,"allocated":417820672,"data_stored":416768220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1026946204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,4,17,22,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":131,"apply_latency_ms":131,"commit_latency_ns":131000000,"apply_latency_ns":131000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7479296,"num_objects":-913,"num_object_clones":0,"num_object_copies":-3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.105897"},"pg_stats":[{"pgid":"1.f","version":"50'7908","reported_seq":"8176","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.416371+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:45.416371+0000","last_peered":"2021-01-15T18:48:45.416371+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:45.416371+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7908,"ondisk_log_size":7908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7933","reported_seq":"10867","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.735729+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:45.735729+0000","last_peered":"2021-01-15T18:48:45.735729+0000","last_clean":"2021-01-15T18:48:45.735729+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:45.735729+0000","last_undegraded":"2021-01-15T18:48:45.735729+0000","last_fullsized":"2021-01-15T18:48:45.735729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7933,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12517542,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7949","reported_seq":"10890","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:43.655676+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:43.655676+0000","last_peered":"2021-01-15T18:48:43.655676+0000","last_clean":"2021-01-15T18:48:43.655676+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:43.655676+0000","last_undegraded":"2021-01-15T18:48:43.655676+0000","last_fullsized":"2021-01-15T18:48:43.655676+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7949,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12636267,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7948,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8204","reported_seq":"8460","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.537706+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:45.537706+0000","last_peered":"2021-01-15T18:48:45.537706+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:45.537706+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8204,"ondisk_log_size":8204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13008896,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8248","reported_seq":"8850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.451168+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:45.451168+0000","last_peered":"2021-01-15T18:48:45.451168+0000","last_clean":"2021-01-15T18:48:45.451168+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:45.451168+0000","last_undegraded":"2021-01-15T18:48:45.451168+0000","last_fullsized":"2021-01-15T18:48:45.451168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8248,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12943526,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7992","reported_seq":"8213","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.429910+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:45.429910+0000","last_peered":"2021-01-15T18:48:45.429910+0000","last_clean":"2021-01-15T18:48:45.429910+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:45.429910+0000","last_undegraded":"2021-01-15T18:48:45.429910+0000","last_fullsized":"2021-01-15T18:48:45.429910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7992,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"50'8089","reported_seq":"10960","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.681147+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:45.681147+0000","last_peered":"2021-01-15T18:48:45.681147+0000","last_clean":"2021-01-15T18:48:45.681147+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:45.681147+0000","last_undegraded":"2021-01-15T18:48:45.681147+0000","last_fullsized":"2021-01-15T18:48:45.681147+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8089,"ondisk_log_size":8089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13402278,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8087,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8187","reported_seq":"11082","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:44.071393+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:44.071393+0000","last_peered":"2021-01-15T18:48:44.071393+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:44.071393+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13676627,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8187","reported_seq":"11132","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.617378+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:45.617378+0000","last_peered":"2021-01-15T18:48:45.617378+0000","last_clean":"2021-01-15T18:48:45.617378+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:45.617378+0000","last_undegraded":"2021-01-15T18:48:45.617378+0000","last_fullsized":"2021-01-15T18:48:45.617378+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12628051,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8170","reported_seq":"11121","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.495582+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:45.495582+0000","last_peered":"2021-01-15T18:48:45.495582+0000","last_clean":"2021-01-15T18:48:45.495582+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:45.495582+0000","last_undegraded":"2021-01-15T18:48:45.495582+0000","last_fullsized":"2021-01-15T18:48:45.495582+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8170,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13005049,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7973","reported_seq":"8223","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.576396+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:45.576396+0000","last_peered":"2021-01-15T18:48:45.576396+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:45.576396+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7973,"ondisk_log_size":7973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12951718,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7971,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'8005","reported_seq":"10843","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.826576+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:45.826576+0000","last_peered":"2021-01-15T18:48:45.826576+0000","last_clean":"2021-01-15T18:48:45.826576+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:45.826576+0000","last_undegraded":"2021-01-15T18:48:45.826576+0000","last_fullsized":"2021-01-15T18:48:45.826576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8005,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12796070,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7906","reported_seq":"10810","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:44.623500+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:44.623500+0000","last_peered":"2021-01-15T18:48:44.623500+0000","last_clean":"2021-01-15T18:48:44.623500+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:44.623500+0000","last_undegraded":"2021-01-15T18:48:44.623500+0000","last_fullsized":"2021-01-15T18:48:44.623500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7906,"ondisk_log_size":7906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12853414,"num_objects":1570,"num_object_clones":0,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8330","reported_seq":"8566","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.668723+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:45.668723+0000","last_peered":"2021-01-15T18:48:45.668723+0000","last_clean":"2021-01-15T18:48:45.668723+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:45.668723+0000","last_undegraded":"2021-01-15T18:48:45.668723+0000","last_fullsized":"2021-01-15T18:48:45.668723+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8330,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12746918,"num_objects":1557,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8424","reported_seq":"8700","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.501551+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:45.501551+0000","last_peered":"2021-01-15T18:48:45.501551+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:45.501551+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8424,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13254988,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7972","reported_seq":"10847","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:44.126051+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:44.126051+0000","last_peered":"2021-01-15T18:48:44.126051+0000","last_clean":"2021-01-15T18:48:44.126051+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:44.126051+0000","last_undegraded":"2021-01-15T18:48:44.126051+0000","last_fullsized":"2021-01-15T18:48:44.126051+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7972,"ondisk_log_size":7972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12837030,"num_objects":1568,"num_object_clones":0,"num_object_copies":6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":206932268,"num_objects":25275,"num_object_clones":0,"num_object_copies":101100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129450,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16091,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414994432,"data_stored":414994432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129477,"ondisk_log_size":129477,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349236,"kb_used_data":101420,"kb_used_omap":0,"kb_used_meta":247807,"kb_avail":94022604,"statfs":{"total":96636764160,"available":96279146496,"internally_reserved":0,"allocated":103854080,"data_stored":103590967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":253754420},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":6.8609999999999998},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":7.452},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":7.508}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.2530000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":7.048}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362432,"kb_used_data":102768,"kb_used_omap":0,"kb_used_meta":259648,"kb_avail":94009408,"statfs":{"total":96636764160,"available":96265633792,"internally_reserved":0,"allocated":105234432,"data_stored":104971319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":265879552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.623},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.75}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.105},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":1.2270000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358032,"kb_used_data":102316,"kb_used_omap":0,"kb_used_meta":255679,"kb_avail":94013808,"statfs":{"total":96636764160,"available":96270139392,"internally_reserved":0,"allocated":104771584,"data_stored":104508471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261815348},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":11.605},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":11.702999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":3.2519999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":3.4289999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.75800000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341300,"kb_used_data":101524,"kb_used_omap":0,"kb_used_meta":239743,"kb_avail":94030540,"statfs":{"total":96636764160,"available":96287272960,"internally_reserved":0,"allocated":103960576,"data_stored":103697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245496884},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":5.1050000000000004},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.0900000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":2.569},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":1.1890000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.2149999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.3069999999999999}]}]}]}} 2021-01-15T18:48:49.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:49.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:49.887 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5316/101100 objects degraded (5.258%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":580,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25275,"data_bytes":206932268,"bytes_used":1444864000,"bytes_avail":385102192640,"bytes_total":386547056640,"degraded_objects":5316,"degraded_total":101100,"degraded_ratio":0.052581602373887239,"recovering_objects_per_sec":80,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":91},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2021-01-15T18:48:46.626754+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:49.900 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:49.900 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:50.389 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:50.399 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2021-01-15T18:48:48.731281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":206932268,"num_objects":25275,"num_object_clones":0,"num_object_copies":101100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129450,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16091,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129477,"ondisk_log_size":129477,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1411000,"kb_used_data":408028,"kb_used_omap":2,"kb_used_meta":1002877,"kb_avail":376076360,"statfs":{"total":386547056640,"available":385102192640,"internally_reserved":0,"allocated":417820672,"data_stored":416768220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1026946204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,4,17,22,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":131,"apply_latency_ms":131,"commit_latency_ns":131000000,"apply_latency_ns":131000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7479296,"num_objects":-913,"num_object_clones":0,"num_object_copies":-3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.105897"},"pg_stats":[{"pgid":"1.f","version":"50'7908","reported_seq":"8176","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.416371+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:45.416371+0000","last_peered":"2021-01-15T18:48:45.416371+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:45.416371+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7908,"ondisk_log_size":7908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13107200,"num_objects":1600,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"50'7933","reported_seq":"10867","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.735729+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:45.735729+0000","last_peered":"2021-01-15T18:48:45.735729+0000","last_clean":"2021-01-15T18:48:45.735729+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:45.735729+0000","last_undegraded":"2021-01-15T18:48:45.735729+0000","last_fullsized":"2021-01-15T18:48:45.735729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7933,"ondisk_log_size":7933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12517542,"num_objects":1529,"num_object_clones":0,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"50'7949","reported_seq":"10890","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:43.655676+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:43.655676+0000","last_peered":"2021-01-15T18:48:43.655676+0000","last_clean":"2021-01-15T18:48:43.655676+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:43.655676+0000","last_undegraded":"2021-01-15T18:48:43.655676+0000","last_fullsized":"2021-01-15T18:48:43.655676+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7949,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12636267,"num_objects":1544,"num_object_clones":0,"num_object_copies":6176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7948,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"50'8204","reported_seq":"8460","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.537706+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:45.537706+0000","last_peered":"2021-01-15T18:48:45.537706+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:45.537706+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8204,"ondisk_log_size":8204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13008896,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"50'8248","reported_seq":"8850","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.451168+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:45.451168+0000","last_peered":"2021-01-15T18:48:45.451168+0000","last_clean":"2021-01-15T18:48:45.451168+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:45.451168+0000","last_undegraded":"2021-01-15T18:48:45.451168+0000","last_fullsized":"2021-01-15T18:48:45.451168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8248,"ondisk_log_size":8248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12943526,"num_objects":1581,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"50'7992","reported_seq":"8213","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.429910+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:45.429910+0000","last_peered":"2021-01-15T18:48:45.429910+0000","last_clean":"2021-01-15T18:48:45.429910+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:45.429910+0000","last_undegraded":"2021-01-15T18:48:45.429910+0000","last_fullsized":"2021-01-15T18:48:45.429910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7992,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"50'8089","reported_seq":"10960","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.681147+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:45.681147+0000","last_peered":"2021-01-15T18:48:45.681147+0000","last_clean":"2021-01-15T18:48:45.681147+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:45.681147+0000","last_undegraded":"2021-01-15T18:48:45.681147+0000","last_fullsized":"2021-01-15T18:48:45.681147+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8089,"ondisk_log_size":8089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13402278,"num_objects":1637,"num_object_clones":0,"num_object_copies":6548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8087,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8187","reported_seq":"11082","reported_epoch":"50","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:44.071393+0000","last_change":"2021-01-15T18:48:26.710986+0000","last_active":"2021-01-15T18:48:44.071393+0000","last_peered":"2021-01-15T18:48:44.071393+0000","last_clean":"2021-01-15T18:39:28.600637+0000","last_became_active":"2021-01-15T18:40:32.773621+0000","last_became_peered":"2021-01-15T18:40:32.773621+0000","last_unstale":"2021-01-15T18:48:44.071393+0000","last_undegraded":"2021-01-15T18:40:32.531198+0000","last_fullsized":"2021-01-15T18:40:32.488431+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13676627,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1404,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"50'8187","reported_seq":"11132","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.617378+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:45.617378+0000","last_peered":"2021-01-15T18:48:45.617378+0000","last_clean":"2021-01-15T18:48:45.617378+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:45.617378+0000","last_undegraded":"2021-01-15T18:48:45.617378+0000","last_fullsized":"2021-01-15T18:48:45.617378+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8187,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12628051,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"50'8170","reported_seq":"11121","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.495582+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:45.495582+0000","last_peered":"2021-01-15T18:48:45.495582+0000","last_clean":"2021-01-15T18:48:45.495582+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:45.495582+0000","last_undegraded":"2021-01-15T18:48:45.495582+0000","last_fullsized":"2021-01-15T18:48:45.495582+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8170,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13005049,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"50'7973","reported_seq":"8223","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.576396+0000","last_change":"2021-01-15T18:40:32.767821+0000","last_active":"2021-01-15T18:48:45.576396+0000","last_peered":"2021-01-15T18:48:45.576396+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:45.576396+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7973,"ondisk_log_size":7973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12951718,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7971,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"50'8005","reported_seq":"10843","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.826576+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:45.826576+0000","last_peered":"2021-01-15T18:48:45.826576+0000","last_clean":"2021-01-15T18:48:45.826576+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:45.826576+0000","last_undegraded":"2021-01-15T18:48:45.826576+0000","last_fullsized":"2021-01-15T18:48:45.826576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8005,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12796070,"num_objects":1563,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8003,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"50'7906","reported_seq":"10810","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:44.623500+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:44.623500+0000","last_peered":"2021-01-15T18:48:44.623500+0000","last_clean":"2021-01-15T18:48:44.623500+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:44.623500+0000","last_undegraded":"2021-01-15T18:48:44.623500+0000","last_fullsized":"2021-01-15T18:48:44.623500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7906,"ondisk_log_size":7906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12853414,"num_objects":1570,"num_object_clones":0,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"50'8330","reported_seq":"8566","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:45.668723+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:45.668723+0000","last_peered":"2021-01-15T18:48:45.668723+0000","last_clean":"2021-01-15T18:48:45.668723+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:45.668723+0000","last_undegraded":"2021-01-15T18:48:45.668723+0000","last_fullsized":"2021-01-15T18:48:45.668723+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8330,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12746918,"num_objects":1557,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"50'8424","reported_seq":"8700","reported_epoch":"50","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:45.501551+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:45.501551+0000","last_peered":"2021-01-15T18:48:45.501551+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:45.501551+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8424,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13254988,"num_objects":1620,"num_object_clones":0,"num_object_copies":6480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"50'7972","reported_seq":"10847","reported_epoch":"50","state":"active+clean","last_fresh":"2021-01-15T18:48:44.126051+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:44.126051+0000","last_peered":"2021-01-15T18:48:44.126051+0000","last_clean":"2021-01-15T18:48:44.126051+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:44.126051+0000","last_undegraded":"2021-01-15T18:48:44.126051+0000","last_fullsized":"2021-01-15T18:48:44.126051+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7972,"ondisk_log_size":7972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12837030,"num_objects":1568,"num_object_clones":0,"num_object_copies":6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":206932268,"num_objects":25275,"num_object_clones":0,"num_object_copies":101100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129450,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16091,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414994432,"data_stored":414994432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129477,"ondisk_log_size":129477,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349236,"kb_used_data":101420,"kb_used_omap":0,"kb_used_meta":247807,"kb_avail":94022604,"statfs":{"total":96636764160,"available":96279146496,"internally_reserved":0,"allocated":103854080,"data_stored":103590967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":253754420},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2,18,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":6.8609999999999998},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":7.452},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":7.508}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.2530000000000001},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":7.048}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362432,"kb_used_data":102768,"kb_used_omap":0,"kb_used_meta":259648,"kb_avail":94009408,"statfs":{"total":96636764160,"available":96265633792,"internally_reserved":0,"allocated":105234432,"data_stored":104971319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":265879552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.623},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.75}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.105},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":1.2270000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":358032,"kb_used_data":102316,"kb_used_omap":0,"kb_used_meta":255679,"kb_avail":94013808,"statfs":{"total":96636764160,"available":96270139392,"internally_reserved":0,"allocated":104771584,"data_stored":104508471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261815348},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":11.605},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":11.702999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":3.2519999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":3.4289999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.75800000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341300,"kb_used_data":101524,"kb_used_omap":0,"kb_used_meta":239743,"kb_avail":94030540,"statfs":{"total":96636764160,"available":96287272960,"internally_reserved":0,"allocated":103960576,"data_stored":103697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245496884},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,9,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":4.4509999999999996,"15min":3.4820000000000002},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":30.905999999999999,"5min":30.905999999999999,"15min":35.850999999999999},"last":5.1050000000000004},{"interface":"front","average":{"1min":2.4809999999999999,"5min":2.3540000000000001,"15min":2.0659999999999998},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.39500000000000002},"max":{"1min":17.263999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.0900000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":2.9769999999999999,"5min":2.6549999999999998,"15min":2.6709999999999998},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":9.6649999999999991,"5min":21.791,"15min":21.791},"last":2.569},{"interface":"front","average":{"1min":3.411,"5min":2.5209999999999999,"15min":1.9059999999999999},"min":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":12.609999999999999,"5min":19.381,"15min":19.381},"last":1.1890000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:47:45 2021","interfaces":[{"interface":"back","average":{"1min":3.476,"5min":3.2349999999999999,"15min":3.7679999999999998},"min":{"1min":0.64500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":17.382999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.2149999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.8650000000000002,"15min":2.6440000000000001},"min":{"1min":0.745,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":17.478000000000002,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.3069999999999999}]}]}]}} 2021-01-15T18:48:53.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:54.127 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:54.138 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2021-01-15T18:48:52.628725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":201517356,"num_objects":24614,"num_object_clones":0,"num_object_copies":98456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24614,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130111,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16236,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130138,"ondisk_log_size":130138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1400580,"kb_used_data":395484,"kb_used_omap":2,"kb_used_meta":1005053,"kb_avail":376086780,"statfs":{"total":386547056640,"available":385112862720,"internally_reserved":0,"allocated":404975616,"data_stored":403883128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1029174428},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,5,10,3,1,2,6,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":104,"apply_latency_ms":104,"commit_latency_ns":104000000,"apply_latency_ns":104000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11919360,"num_objects":-1455,"num_object_clones":0,"num_object_copies":-5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1455,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002500"},"pg_stats":[{"pgid":"1.f","version":"52'7943","reported_seq":"8213","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.342897+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:49.342897+0000","last_peered":"2021-01-15T18:48:49.342897+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:49.342897+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7943,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12820480,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'7977","reported_seq":"10913","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.449426+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:49.449426+0000","last_peered":"2021-01-15T18:48:49.449426+0000","last_clean":"2021-01-15T18:48:49.449426+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:49.449426+0000","last_undegraded":"2021-01-15T18:48:49.449426+0000","last_fullsized":"2021-01-15T18:48:49.449426+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7977,"ondisk_log_size":7977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12157094,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8005","reported_seq":"10947","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.079744+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:49.079744+0000","last_peered":"2021-01-15T18:48:49.079744+0000","last_clean":"2021-01-15T18:48:49.079744+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:49.079744+0000","last_undegraded":"2021-01-15T18:48:49.079744+0000","last_fullsized":"2021-01-15T18:48:49.079744+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8005,"ondisk_log_size":8005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12177515,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8004,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8245","reported_seq":"8503","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.350757+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:49.350757+0000","last_peered":"2021-01-15T18:48:49.350757+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:49.350757+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8245,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12673024,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8288","reported_seq":"8892","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.648921+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:49.648921+0000","last_peered":"2021-01-15T18:48:49.648921+0000","last_clean":"2021-01-15T18:48:49.648921+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:49.648921+0000","last_undegraded":"2021-01-15T18:48:49.648921+0000","last_fullsized":"2021-01-15T18:48:49.648921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8288,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12615846,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8037","reported_seq":"8260","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.632734+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:49.632734+0000","last_peered":"2021-01-15T18:48:49.632734+0000","last_clean":"2021-01-15T18:48:49.632734+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:49.632734+0000","last_undegraded":"2021-01-15T18:48:49.632734+0000","last_fullsized":"2021-01-15T18:48:49.632734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8037,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12198054,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8131","reported_seq":"11004","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.602895+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:49.602895+0000","last_peered":"2021-01-15T18:48:49.602895+0000","last_clean":"2021-01-15T18:48:49.602895+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:49.602895+0000","last_undegraded":"2021-01-15T18:48:49.602895+0000","last_fullsized":"2021-01-15T18:48:49.602895+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8131,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13058214,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8235","reported_seq":"11243","reported_epoch":"52","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:48.950126+0000","last_change":"2021-01-15T18:48:48.950126+0000","last_active":"2021-01-15T18:48:48.950126+0000","last_peered":"2021-01-15T18:48:48.950126+0000","last_clean":"2021-01-15T18:48:48.950126+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:48:48.950126+0000","last_undegraded":"2021-01-15T18:48:48.950126+0000","last_fullsized":"2021-01-15T18:48:48.950126+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8235,"ondisk_log_size":8235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13283411,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8223","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.009469+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:49.009469+0000","last_peered":"2021-01-15T18:48:49.009469+0000","last_clean":"2021-01-15T18:48:49.009469+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:49.009469+0000","last_undegraded":"2021-01-15T18:48:49.009469+0000","last_fullsized":"2021-01-15T18:48:49.009469+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8223,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12333139,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8217","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.763711+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:49.763711+0000","last_peered":"2021-01-15T18:48:49.763711+0000","last_clean":"2021-01-15T18:48:49.763711+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:49.763711+0000","last_undegraded":"2021-01-15T18:48:49.763711+0000","last_fullsized":"2021-01-15T18:48:49.763711+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8217,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12620025,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8008","reported_seq":"8451","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:50.642903+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:48:50.642903+0000","last_peered":"2021-01-15T18:48:50.642903+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:50.642903+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12664998,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8033","reported_seq":"10874","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.580876+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:49.580876+0000","last_peered":"2021-01-15T18:48:49.580876+0000","last_clean":"2021-01-15T18:48:49.580876+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:49.580876+0000","last_undegraded":"2021-01-15T18:48:49.580876+0000","last_fullsized":"2021-01-15T18:48:49.580876+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8033,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"51'7951","reported_seq":"10858","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:48.770930+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:48.770930+0000","last_peered":"2021-01-15T18:48:48.770930+0000","last_clean":"2021-01-15T18:48:48.770930+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:48.770930+0000","last_undegraded":"2021-01-15T18:48:48.770930+0000","last_fullsized":"2021-01-15T18:48:48.770930+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7951,"ondisk_log_size":7951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12484774,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7949,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8367","reported_seq":"8605","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.214818+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:49.214818+0000","last_peered":"2021-01-15T18:48:49.214818+0000","last_clean":"2021-01-15T18:48:49.214818+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:49.214818+0000","last_undegraded":"2021-01-15T18:48:49.214818+0000","last_fullsized":"2021-01-15T18:48:49.214818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8367,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12443814,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8465","reported_seq":"8743","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.384953+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:49.384953+0000","last_peered":"2021-01-15T18:48:49.384953+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:49.384953+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8465,"ondisk_log_size":8465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12919116,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8013","reported_seq":"10890","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:48.840335+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:48.840335+0000","last_peered":"2021-01-15T18:48:48.840335+0000","last_clean":"2021-01-15T18:48:48.840335+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:48.840335+0000","last_undegraded":"2021-01-15T18:48:48.840335+0000","last_fullsized":"2021-01-15T18:48:48.840335+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8013,"ondisk_log_size":8013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12501158,"num_objects":1527,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":201517356,"num_objects":24614,"num_object_clones":0,"num_object_copies":98456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24614,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130111,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16236,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402964480,"data_stored":402964480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130138,"ondisk_log_size":130138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356084,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":257343,"kb_avail":94015756,"statfs":{"total":96636764160,"available":96272134144,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263519284},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.9630000000000001},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":3.6789999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":5.1609999999999996},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":5.2430000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":5.0709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348212,"kb_used_data":99108,"kb_used_omap":0,"kb_used_meta":249088,"kb_avail":94023628,"statfs":{"total":96636764160,"available":96280195072,"internally_reserved":0,"allocated":101486592,"data_stored":101213470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":255066112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.90400000000000003},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.0820000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.1180000000000001},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.69499999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":3.706},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.65800000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348052,"kb_used_data":98892,"kb_used_omap":0,"kb_used_meta":249151,"kb_avail":94023788,"statfs":{"total":96636764160,"available":96280358912,"internally_reserved":0,"allocated":101265408,"data_stored":100992286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255130676},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.442},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.622},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.67900000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348232,"kb_used_data":98752,"kb_used_omap":0,"kb_used_meta":249471,"kb_avail":94023608,"statfs":{"total":96636764160,"available":96280174592,"internally_reserved":0,"allocated":101122048,"data_stored":100848926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255458356},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,0,1,2,4,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":4.7789999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.292},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0009999999999999}]}]}]}} 2021-01-15T18:48:54.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:48:54.490 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:54.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5171/98456 objects degraded (5.252%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":584,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24614,"data_bytes":201517356,"bytes_used":1434193920,"bytes_avail":385112862720,"bytes_total":386547056640,"degraded_objects":5171,"degraded_total":98456,"degraded_ratio":0.052520923051921665,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2021-01-15T18:48:46.626754+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:48:54.506 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:48:54.506 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:54.816 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:54.817 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:54.829 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2021-01-15T18:48:54.629209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":201443628,"num_objects":24605,"num_object_clones":0,"num_object_copies":98420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24605,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130120,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16236,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130147,"ondisk_log_size":130147,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1404548,"kb_used_data":395108,"kb_used_omap":2,"kb_used_meta":1009405,"kb_avail":376082812,"statfs":{"total":386547056640,"available":385108799488,"internally_reserved":0,"allocated":404590592,"data_stored":403498104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1033630876},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2,8,0,1,2,6,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":86,"apply_latency_ms":86,"commit_latency_ns":86000000,"apply_latency_ns":86000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5488640,"num_objects":-670,"num_object_clones":0,"num_object_copies":-2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":670,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002553"},"pg_stats":[{"pgid":"1.f","version":"52'7943","reported_seq":"8213","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.342897+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:49.342897+0000","last_peered":"2021-01-15T18:48:49.342897+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:49.342897+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7943,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12820480,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'7977","reported_seq":"10913","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.449426+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:49.449426+0000","last_peered":"2021-01-15T18:48:49.449426+0000","last_clean":"2021-01-15T18:48:49.449426+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:49.449426+0000","last_undegraded":"2021-01-15T18:48:49.449426+0000","last_fullsized":"2021-01-15T18:48:49.449426+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7977,"ondisk_log_size":7977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12157094,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8008","reported_seq":"10951","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.598062+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:49.598062+0000","last_peered":"2021-01-15T18:48:49.598062+0000","last_clean":"2021-01-15T18:48:49.598062+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:49.598062+0000","last_undegraded":"2021-01-15T18:48:49.598062+0000","last_fullsized":"2021-01-15T18:48:49.598062+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12152939,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8007,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8245","reported_seq":"8503","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.350757+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:49.350757+0000","last_peered":"2021-01-15T18:48:49.350757+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:49.350757+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8245,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12673024,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8288","reported_seq":"8892","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.648921+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:49.648921+0000","last_peered":"2021-01-15T18:48:49.648921+0000","last_clean":"2021-01-15T18:48:49.648921+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:49.648921+0000","last_undegraded":"2021-01-15T18:48:49.648921+0000","last_fullsized":"2021-01-15T18:48:49.648921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8288,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12615846,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8037","reported_seq":"8260","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.632734+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:49.632734+0000","last_peered":"2021-01-15T18:48:49.632734+0000","last_clean":"2021-01-15T18:48:49.632734+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:49.632734+0000","last_undegraded":"2021-01-15T18:48:49.632734+0000","last_fullsized":"2021-01-15T18:48:49.632734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8037,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12198054,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8131","reported_seq":"11004","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.602895+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:49.602895+0000","last_peered":"2021-01-15T18:48:49.602895+0000","last_clean":"2021-01-15T18:48:49.602895+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:49.602895+0000","last_undegraded":"2021-01-15T18:48:49.602895+0000","last_fullsized":"2021-01-15T18:48:49.602895+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8131,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13058214,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8235","reported_seq":"11243","reported_epoch":"52","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:48.950126+0000","last_change":"2021-01-15T18:48:48.950126+0000","last_active":"2021-01-15T18:48:48.950126+0000","last_peered":"2021-01-15T18:48:48.950126+0000","last_clean":"2021-01-15T18:48:48.950126+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:48:48.950126+0000","last_undegraded":"2021-01-15T18:48:48.950126+0000","last_fullsized":"2021-01-15T18:48:48.950126+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8235,"ondisk_log_size":8235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13283411,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8223","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.009469+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:49.009469+0000","last_peered":"2021-01-15T18:48:49.009469+0000","last_clean":"2021-01-15T18:48:49.009469+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:49.009469+0000","last_undegraded":"2021-01-15T18:48:49.009469+0000","last_fullsized":"2021-01-15T18:48:49.009469+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8223,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12333139,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8217","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.763711+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:49.763711+0000","last_peered":"2021-01-15T18:48:49.763711+0000","last_clean":"2021-01-15T18:48:49.763711+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:49.763711+0000","last_undegraded":"2021-01-15T18:48:49.763711+0000","last_fullsized":"2021-01-15T18:48:49.763711+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8217,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12620025,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8008","reported_seq":"8451","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:50.642903+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:48:50.642903+0000","last_peered":"2021-01-15T18:48:50.642903+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:50.642903+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12664998,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8033","reported_seq":"10874","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.580876+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:49.580876+0000","last_peered":"2021-01-15T18:48:49.580876+0000","last_clean":"2021-01-15T18:48:49.580876+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:49.580876+0000","last_undegraded":"2021-01-15T18:48:49.580876+0000","last_fullsized":"2021-01-15T18:48:49.580876+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8033,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"51'7951","reported_seq":"10858","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:48.770930+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:48.770930+0000","last_peered":"2021-01-15T18:48:48.770930+0000","last_clean":"2021-01-15T18:48:48.770930+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:48.770930+0000","last_undegraded":"2021-01-15T18:48:48.770930+0000","last_fullsized":"2021-01-15T18:48:48.770930+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7951,"ondisk_log_size":7951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12484774,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7949,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8367","reported_seq":"8605","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.214818+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:49.214818+0000","last_peered":"2021-01-15T18:48:49.214818+0000","last_clean":"2021-01-15T18:48:49.214818+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:49.214818+0000","last_undegraded":"2021-01-15T18:48:49.214818+0000","last_fullsized":"2021-01-15T18:48:49.214818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8367,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12443814,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8465","reported_seq":"8743","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.384953+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:49.384953+0000","last_peered":"2021-01-15T18:48:49.384953+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:49.384953+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8465,"ondisk_log_size":8465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12919116,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8019","reported_seq":"10896","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.783489+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:49.783489+0000","last_peered":"2021-01-15T18:48:49.783489+0000","last_clean":"2021-01-15T18:48:49.783489+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:49.783489+0000","last_undegraded":"2021-01-15T18:48:49.783489+0000","last_fullsized":"2021-01-15T18:48:49.783489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8019,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12452006,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":201443628,"num_objects":24605,"num_object_clones":0,"num_object_copies":98420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24605,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130120,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16236,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402714624,"data_stored":402714624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130147,"ondisk_log_size":130147,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356084,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":257343,"kb_avail":94015756,"statfs":{"total":96636764160,"available":96272134144,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263519284},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":4.9630000000000001},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":3.6789999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":5.1609999999999996},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":5.2430000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":5.0709999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352180,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":253440,"kb_avail":94019660,"statfs":{"total":96636764160,"available":96276131840,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":259522560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.5}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.747},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.84999999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348052,"kb_used_data":98892,"kb_used_omap":0,"kb_used_meta":249151,"kb_avail":94023788,"statfs":{"total":96636764160,"available":96280358912,"internally_reserved":0,"allocated":101265408,"data_stored":100992286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255130676},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.442},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.622},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.67900000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348232,"kb_used_data":98752,"kb_used_omap":0,"kb_used_meta":249471,"kb_avail":94023608,"statfs":{"total":96636764160,"available":96280174592,"internally_reserved":0,"allocated":101122048,"data_stored":100848926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255458356},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,6,0,1,2,4,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":4.7789999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.292},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0009999999999999}]}]}]}} 2021-01-15T18:48:57.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:48:59.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:48:59.134 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:48:59.155 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2021-01-15T18:48:58.629982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":201435436,"num_objects":24604,"num_object_clones":0,"num_object_copies":98416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130121,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16768,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130148,"ondisk_log_size":130148,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1396900,"kb_used_data":394948,"kb_used_omap":2,"kb_used_meta":1001917,"kb_avail":376090460,"statfs":{"total":386547056640,"available":385116631040,"internally_reserved":0,"allocated":404426752,"data_stored":403334264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1025963164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5496832,"num_objects":-671,"num_object_clones":0,"num_object_copies":-2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002174"},"pg_stats":[{"pgid":"1.f","version":"52'7943","reported_seq":"8213","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.342897+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:49.342897+0000","last_peered":"2021-01-15T18:48:49.342897+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:49.342897+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7943,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12820480,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'7977","reported_seq":"10913","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.449426+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:49.449426+0000","last_peered":"2021-01-15T18:48:49.449426+0000","last_clean":"2021-01-15T18:48:49.449426+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:49.449426+0000","last_undegraded":"2021-01-15T18:48:49.449426+0000","last_fullsized":"2021-01-15T18:48:49.449426+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7977,"ondisk_log_size":7977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12157094,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8008","reported_seq":"10951","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.598062+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:49.598062+0000","last_peered":"2021-01-15T18:48:49.598062+0000","last_clean":"2021-01-15T18:48:49.598062+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:49.598062+0000","last_undegraded":"2021-01-15T18:48:49.598062+0000","last_fullsized":"2021-01-15T18:48:49.598062+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12152939,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8007,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8245","reported_seq":"8503","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.350757+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:49.350757+0000","last_peered":"2021-01-15T18:48:49.350757+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:49.350757+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8245,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12673024,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8288","reported_seq":"8892","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.648921+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:49.648921+0000","last_peered":"2021-01-15T18:48:49.648921+0000","last_clean":"2021-01-15T18:48:49.648921+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:49.648921+0000","last_undegraded":"2021-01-15T18:48:49.648921+0000","last_fullsized":"2021-01-15T18:48:49.648921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8288,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12615846,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8037","reported_seq":"8260","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.632734+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:49.632734+0000","last_peered":"2021-01-15T18:48:49.632734+0000","last_clean":"2021-01-15T18:48:49.632734+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:49.632734+0000","last_undegraded":"2021-01-15T18:48:49.632734+0000","last_fullsized":"2021-01-15T18:48:49.632734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8037,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12198054,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8131","reported_seq":"11004","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.602895+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:49.602895+0000","last_peered":"2021-01-15T18:48:49.602895+0000","last_clean":"2021-01-15T18:48:49.602895+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:49.602895+0000","last_undegraded":"2021-01-15T18:48:49.602895+0000","last_fullsized":"2021-01-15T18:48:49.602895+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8131,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13058214,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8235","reported_seq":"11243","reported_epoch":"52","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:48.950126+0000","last_change":"2021-01-15T18:48:48.950126+0000","last_active":"2021-01-15T18:48:48.950126+0000","last_peered":"2021-01-15T18:48:48.950126+0000","last_clean":"2021-01-15T18:48:48.950126+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:48:48.950126+0000","last_undegraded":"2021-01-15T18:48:48.950126+0000","last_fullsized":"2021-01-15T18:48:48.950126+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8235,"ondisk_log_size":8235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13283411,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8223","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.009469+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:49.009469+0000","last_peered":"2021-01-15T18:48:49.009469+0000","last_clean":"2021-01-15T18:48:49.009469+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:49.009469+0000","last_undegraded":"2021-01-15T18:48:49.009469+0000","last_fullsized":"2021-01-15T18:48:49.009469+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8223,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12333139,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8217","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.763711+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:49.763711+0000","last_peered":"2021-01-15T18:48:49.763711+0000","last_clean":"2021-01-15T18:48:49.763711+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:49.763711+0000","last_undegraded":"2021-01-15T18:48:49.763711+0000","last_fullsized":"2021-01-15T18:48:49.763711+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8217,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12620025,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8008","reported_seq":"9515","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:55.645009+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:48:55.645009+0000","last_peered":"2021-01-15T18:48:55.645009+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:55.645009+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12664998,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8033","reported_seq":"10874","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.580876+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:49.580876+0000","last_peered":"2021-01-15T18:48:49.580876+0000","last_clean":"2021-01-15T18:48:49.580876+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:49.580876+0000","last_undegraded":"2021-01-15T18:48:49.580876+0000","last_fullsized":"2021-01-15T18:48:49.580876+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8033,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'7952","reported_seq":"10859","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:48.782089+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:48.782089+0000","last_peered":"2021-01-15T18:48:48.782089+0000","last_clean":"2021-01-15T18:48:48.782089+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:48.782089+0000","last_undegraded":"2021-01-15T18:48:48.782089+0000","last_fullsized":"2021-01-15T18:48:48.782089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7952,"ondisk_log_size":7952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12476582,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7950,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8367","reported_seq":"8605","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.214818+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:49.214818+0000","last_peered":"2021-01-15T18:48:49.214818+0000","last_clean":"2021-01-15T18:48:49.214818+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:49.214818+0000","last_undegraded":"2021-01-15T18:48:49.214818+0000","last_fullsized":"2021-01-15T18:48:49.214818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8367,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12443814,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8465","reported_seq":"8743","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.384953+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:49.384953+0000","last_peered":"2021-01-15T18:48:49.384953+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:49.384953+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8465,"ondisk_log_size":8465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12919116,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8019","reported_seq":"10896","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.783489+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:49.783489+0000","last_peered":"2021-01-15T18:48:49.783489+0000","last_clean":"2021-01-15T18:48:49.783489+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:49.783489+0000","last_undegraded":"2021-01-15T18:48:49.783489+0000","last_fullsized":"2021-01-15T18:48:49.783489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8019,"ondisk_log_size":8019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12452006,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":201435436,"num_objects":24604,"num_object_clones":0,"num_object_copies":98416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130121,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16768,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402690048,"data_stored":402690048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130148,"ondisk_log_size":130148,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356084,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":257343,"kb_avail":94015756,"statfs":{"total":96636764160,"available":96272134144,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263519284},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.438},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":0.755}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352180,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":253440,"kb_avail":94019660,"statfs":{"total":96636764160,"available":96276131840,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":259522560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.5}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.747},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.84999999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347956,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":249215,"kb_avail":94023884,"statfs":{"total":96636764160,"available":96280457216,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255196212},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.38400000000000001},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.69399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.626},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.74399999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340680,"kb_used_data":98752,"kb_used_omap":0,"kb_used_meta":241919,"kb_avail":94031160,"statfs":{"total":96636764160,"available":96287907840,"internally_reserved":0,"allocated":101122048,"data_stored":100848926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":247725108},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.66400000000000003}]}]}]}} 2021-01-15T18:48:59.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:00.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:00.450 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4639/98416 objects degraded (4.714%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":590,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+wait","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24604,"data_bytes":201435436,"bytes_used":1430425600,"bytes_avail":385116631040,"bytes_total":386547056640,"degraded_objects":4639,"degraded_total":98416,"degraded_ratio":0.047136644448057229,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":677,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":67},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2021-01-15T18:48:46.626754+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:49:00.484 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:00.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:01.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:01.585 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:01.644 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2021-01-15T18:49:00.630381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200788268,"num_objects":24525,"num_object_clones":0,"num_object_copies":98100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130200,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16768,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130227,"ondisk_log_size":130227,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1407376,"kb_used_data":391368,"kb_used_omap":2,"kb_used_meta":1015869,"kb_avail":376079984,"statfs":{"total":386547056640,"available":385105903616,"internally_reserved":0,"allocated":400760832,"data_stored":399668344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1040250012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,7,6,1,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6144000,"num_objects":-750,"num_object_clones":0,"num_object_copies":-3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.899099"},"pg_stats":[{"pgid":"1.f","version":"52'7943","reported_seq":"8213","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.342897+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:48:49.342897+0000","last_peered":"2021-01-15T18:48:49.342897+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:48:49.342897+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7943,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12820480,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'7977","reported_seq":"10913","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.449426+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:48:49.449426+0000","last_peered":"2021-01-15T18:48:49.449426+0000","last_clean":"2021-01-15T18:48:49.449426+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:48:49.449426+0000","last_undegraded":"2021-01-15T18:48:49.449426+0000","last_fullsized":"2021-01-15T18:48:49.449426+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7977,"ondisk_log_size":7977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12157094,"num_objects":1485,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8029","reported_seq":"10972","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:59.063609+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:48:59.063609+0000","last_peered":"2021-01-15T18:48:59.063609+0000","last_clean":"2021-01-15T18:48:59.063609+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:48:59.063609+0000","last_undegraded":"2021-01-15T18:48:59.063609+0000","last_fullsized":"2021-01-15T18:48:59.063609+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8029,"ondisk_log_size":8029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11980907,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8028,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8245","reported_seq":"8503","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.350757+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:48:49.350757+0000","last_peered":"2021-01-15T18:48:49.350757+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:48:49.350757+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8245,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12673024,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":213}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8288","reported_seq":"8892","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.648921+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:48:49.648921+0000","last_peered":"2021-01-15T18:48:49.648921+0000","last_clean":"2021-01-15T18:48:49.648921+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:48:49.648921+0000","last_undegraded":"2021-01-15T18:48:49.648921+0000","last_fullsized":"2021-01-15T18:48:49.648921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8288,"ondisk_log_size":8288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12615846,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8286,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8037","reported_seq":"8260","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.632734+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:48:49.632734+0000","last_peered":"2021-01-15T18:48:49.632734+0000","last_clean":"2021-01-15T18:48:49.632734+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:48:49.632734+0000","last_undegraded":"2021-01-15T18:48:49.632734+0000","last_fullsized":"2021-01-15T18:48:49.632734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8037,"ondisk_log_size":8037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12198054,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8131","reported_seq":"11004","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.602895+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:48:49.602895+0000","last_peered":"2021-01-15T18:48:49.602895+0000","last_clean":"2021-01-15T18:48:49.602895+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:48:49.602895+0000","last_undegraded":"2021-01-15T18:48:49.602895+0000","last_fullsized":"2021-01-15T18:48:49.602895+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8131,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13058214,"num_objects":1595,"num_object_clones":0,"num_object_copies":6380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"50'8235","reported_seq":"11243","reported_epoch":"52","state":"active+clean+wait","last_fresh":"2021-01-15T18:48:48.950126+0000","last_change":"2021-01-15T18:48:48.950126+0000","last_active":"2021-01-15T18:48:48.950126+0000","last_peered":"2021-01-15T18:48:48.950126+0000","last_clean":"2021-01-15T18:48:48.950126+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:48:48.950126+0000","last_undegraded":"2021-01-15T18:48:48.950126+0000","last_fullsized":"2021-01-15T18:48:48.950126+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8235,"ondisk_log_size":8235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13283411,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8223","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.009469+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:48:49.009469+0000","last_peered":"2021-01-15T18:48:49.009469+0000","last_clean":"2021-01-15T18:48:49.009469+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:48:49.009469+0000","last_undegraded":"2021-01-15T18:48:49.009469+0000","last_fullsized":"2021-01-15T18:48:49.009469+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8223,"ondisk_log_size":8223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12333139,"num_objects":1506,"num_object_clones":0,"num_object_copies":6024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8217","reported_seq":"11170","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.763711+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:48:49.763711+0000","last_peered":"2021-01-15T18:48:49.763711+0000","last_clean":"2021-01-15T18:48:49.763711+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:48:49.763711+0000","last_undegraded":"2021-01-15T18:48:49.763711+0000","last_fullsized":"2021-01-15T18:48:49.763711+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8217,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12620025,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8008","reported_seq":"9515","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:55.645009+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:48:55.645009+0000","last_peered":"2021-01-15T18:48:55.645009+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:48:55.645009+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8008,"ondisk_log_size":8008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12664998,"num_objects":1547,"num_object_clones":0,"num_object_copies":6188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8006,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8033","reported_seq":"10874","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.580876+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:48:49.580876+0000","last_peered":"2021-01-15T18:48:49.580876+0000","last_clean":"2021-01-15T18:48:49.580876+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:48:49.580876+0000","last_undegraded":"2021-01-15T18:48:49.580876+0000","last_fullsized":"2021-01-15T18:48:49.580876+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8033,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12566694,"num_objects":1535,"num_object_clones":0,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'7981","reported_seq":"10888","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:59.580300+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:59.580300+0000","last_peered":"2021-01-15T18:48:59.580300+0000","last_clean":"2021-01-15T18:48:59.580300+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:59.580300+0000","last_undegraded":"2021-01-15T18:48:59.580300+0000","last_fullsized":"2021-01-15T18:48:59.580300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7981,"ondisk_log_size":7981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12239014,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8367","reported_seq":"8605","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:49.214818+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:48:49.214818+0000","last_peered":"2021-01-15T18:48:49.214818+0000","last_clean":"2021-01-15T18:48:49.214818+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:48:49.214818+0000","last_undegraded":"2021-01-15T18:48:49.214818+0000","last_fullsized":"2021-01-15T18:48:49.214818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8367,"ondisk_log_size":8367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12443814,"num_objects":1520,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8365,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8465","reported_seq":"8743","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:48:49.384953+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:48:49.384953+0000","last_peered":"2021-01-15T18:48:49.384953+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:48:49.384953+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8465,"ondisk_log_size":8465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12919116,"num_objects":1579,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8048","reported_seq":"10925","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:58.959827+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:48:58.959827+0000","last_peered":"2021-01-15T18:48:58.959827+0000","last_clean":"2021-01-15T18:48:58.959827+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:48:58.959827+0000","last_undegraded":"2021-01-15T18:48:58.959827+0000","last_fullsized":"2021-01-15T18:48:58.959827+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8048,"ondisk_log_size":8048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12214438,"num_objects":1492,"num_object_clones":0,"num_object_copies":5968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":200788268,"num_objects":24525,"num_object_clones":0,"num_object_copies":98100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130200,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16768,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":398540800,"data_stored":398540800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130227,"ondisk_log_size":130227,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215217,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356084,"kb_used_data":98732,"kb_used_omap":0,"kb_used_meta":257343,"kb_avail":94015756,"statfs":{"total":96636764160,"available":96272134144,"internally_reserved":0,"allocated":101101568,"data_stored":100828446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":263519284},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.438},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":0.755}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355556,"kb_used_data":97208,"kb_used_omap":0,"kb_used_meta":258240,"kb_avail":94016284,"statfs":{"total":96636764160,"available":96272674816,"internally_reserved":0,"allocated":99540992,"data_stored":99267870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":264437760},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":3.169,"15min":3.5139999999999998},"min":{"1min":0.43099999999999999,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":16.923999999999999,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":3.7549999999999999,"5min":2.9159999999999999,"15min":1.859},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":27.861999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.50900000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.63}]},{"osd":3,"last update":"Fri Jan 15 18:47:57 2021","interfaces":[{"interface":"back","average":{"1min":3.488,"5min":3.484,"15min":2.758},"min":{"1min":0.59599999999999997,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":16.715,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.339,"5min":2.9249999999999998,"15min":2.6080000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":16.821999999999999,"5min":18.34,"15min":18.34},"last":0.70699999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355056,"kb_used_data":96676,"kb_used_omap":0,"kb_used_meta":258367,"kb_avail":94016784,"statfs":{"total":96636764160,"available":96273186816,"internally_reserved":0,"allocated":98996224,"data_stored":98723102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264567860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.60499999999999998},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":4.9950000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":1.0369999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":9.6980000000000004},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.67400000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340680,"kb_used_data":98752,"kb_used_omap":0,"kb_used_meta":241919,"kb_avail":94031160,"statfs":{"total":96636764160,"available":96287907840,"internally_reserved":0,"allocated":101122048,"data_stored":100848926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":247725108},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,1,19],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.66400000000000003}]}]}]}} 2021-01-15T18:49:04.647 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:06.163 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:06.165 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:06.386 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2021-01-15T18:49:04.633807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194627884,"num_objects":23773,"num_object_clones":0,"num_object_copies":95092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130952,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16954,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130979,"ondisk_log_size":130979,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1377864,"kb_used_data":381756,"kb_used_omap":2,"kb_used_meta":995901,"kb_avail":376109496,"statfs":{"total":386547056640,"available":385136123904,"internally_reserved":0,"allocated":390918144,"data_stored":389825656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1019802780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,11,19,15,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":121,"apply_latency_ms":121,"commit_latency_ns":121000000,"apply_latency_ns":121000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6889472,"num_objects":-841,"num_object_clones":0,"num_object_copies":-3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":841,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005081"},"pg_stats":[{"pgid":"1.f","version":"52'7984","reported_seq":"8254","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:00.268964+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:00.268964+0000","last_peered":"2021-01-15T18:49:00.268964+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:00.268964+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7984,"ondisk_log_size":7984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12484608,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7984,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8027","reported_seq":"10963","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.516144+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:00.516144+0000","last_peered":"2021-01-15T18:49:00.516144+0000","last_clean":"2021-01-15T18:49:00.516144+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:00.516144+0000","last_undegraded":"2021-01-15T18:49:00.516144+0000","last_fullsized":"2021-01-15T18:49:00.516144+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8027,"ondisk_log_size":8027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11747494,"num_objects":1435,"num_object_clones":0,"num_object_copies":5740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8025,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8086","reported_seq":"11027","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:04.099594+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:04.099594+0000","last_peered":"2021-01-15T18:49:04.099594+0000","last_clean":"2021-01-15T18:49:04.099594+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:04.099594+0000","last_undegraded":"2021-01-15T18:49:04.099594+0000","last_fullsized":"2021-01-15T18:49:04.099594+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8086,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11513963,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8085,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8298","reported_seq":"8555","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:00.437798+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:00.437798+0000","last_peered":"2021-01-15T18:49:00.437798+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:00.437798+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8298,"ondisk_log_size":8298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12238848,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8298,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":160}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8330","reported_seq":"8934","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.696773+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:00.696773+0000","last_peered":"2021-01-15T18:49:00.696773+0000","last_clean":"2021-01-15T18:49:00.696773+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:00.696773+0000","last_undegraded":"2021-01-15T18:49:00.696773+0000","last_fullsized":"2021-01-15T18:49:00.696773+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8330,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12271782,"num_objects":1499,"num_object_clones":0,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8081","reported_seq":"8304","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.742338+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:00.742338+0000","last_peered":"2021-01-15T18:49:00.742338+0000","last_clean":"2021-01-15T18:49:00.742338+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:00.742338+0000","last_undegraded":"2021-01-15T18:49:00.742338+0000","last_fullsized":"2021-01-15T18:49:00.742338+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8081,"ondisk_log_size":8081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11837606,"num_objects":1446,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8185","reported_seq":"11057","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.687660+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:00.687660+0000","last_peered":"2021-01-15T18:49:00.687660+0000","last_clean":"2021-01-15T18:49:00.687660+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:00.687660+0000","last_undegraded":"2021-01-15T18:49:00.687660+0000","last_fullsized":"2021-01-15T18:49:00.687660+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8185,"ondisk_log_size":8185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12615846,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8304","reported_seq":"11313","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.666225+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:00.666225+0000","last_peered":"2021-01-15T18:49:00.666225+0000","last_clean":"2021-01-15T18:49:00.666225+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:00.666225+0000","last_undegraded":"2021-01-15T18:49:00.666225+0000","last_fullsized":"2021-01-15T18:49:00.666225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8304,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12718163,"num_objects":1553,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8268","reported_seq":"11214","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.802155+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:00.802155+0000","last_peered":"2021-01-15T18:49:00.802155+0000","last_clean":"2021-01-15T18:49:00.802155+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:00.802155+0000","last_undegraded":"2021-01-15T18:49:00.802155+0000","last_fullsized":"2021-01-15T18:49:00.802155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8268,"ondisk_log_size":8268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11964499,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8257","reported_seq":"11210","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.386525+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:00.386525+0000","last_peered":"2021-01-15T18:49:00.386525+0000","last_clean":"2021-01-15T18:49:00.386525+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:00.386525+0000","last_undegraded":"2021-01-15T18:49:00.386525+0000","last_fullsized":"2021-01-15T18:49:00.386525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8257,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12292345,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8254,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8058","reported_seq":"9936","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:00.676680+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:00.676680+0000","last_peered":"2021-01-15T18:49:00.676680+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:00.676680+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8058,"ondisk_log_size":8058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12255398,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1016}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8086","reported_seq":"10927","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.563738+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:00.563738+0000","last_peered":"2021-01-15T18:49:00.563738+0000","last_clean":"2021-01-15T18:49:00.563738+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:00.563738+0000","last_undegraded":"2021-01-15T18:49:00.563738+0000","last_fullsized":"2021-01-15T18:49:00.563738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8086,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12132518,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8084,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'7981","reported_seq":"10888","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:48:59.580300+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:48:59.580300+0000","last_peered":"2021-01-15T18:48:59.580300+0000","last_clean":"2021-01-15T18:48:59.580300+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:48:59.580300+0000","last_undegraded":"2021-01-15T18:48:59.580300+0000","last_fullsized":"2021-01-15T18:48:59.580300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":7981,"ondisk_log_size":7981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12239014,"num_objects":1495,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8420","reported_seq":"8658","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:00.751015+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:00.751015+0000","last_peered":"2021-01-15T18:49:00.751015+0000","last_clean":"2021-01-15T18:49:00.751015+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:00.751015+0000","last_undegraded":"2021-01-15T18:49:00.751015+0000","last_fullsized":"2021-01-15T18:49:00.751015+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8420,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12009638,"num_objects":1467,"num_object_clones":0,"num_object_copies":5868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8418,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8516","reported_seq":"8794","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:00.563269+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:00.563269+0000","last_peered":"2021-01-15T18:49:00.563269+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:00.563269+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8516,"ondisk_log_size":8516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12501324,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8513,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8098","reported_seq":"10975","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:04.099146+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:04.099146+0000","last_peered":"2021-01-15T18:49:04.099146+0000","last_clean":"2021-01-15T18:49:04.099146+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:04.099146+0000","last_undegraded":"2021-01-15T18:49:04.099146+0000","last_fullsized":"2021-01-15T18:49:04.099146+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8098,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11804838,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8096,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194627884,"num_objects":23773,"num_object_clones":0,"num_object_copies":95092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130952,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16954,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":388161536,"data_stored":388161536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130979,"ondisk_log_size":130979,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215218,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342312,"kb_used_data":95888,"kb_used_omap":0,"kb_used_meta":246335,"kb_avail":94029528,"statfs":{"total":96636764160,"available":96286236672,"internally_reserved":0,"allocated":98189312,"data_stored":97916190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":252247092},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,0,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":8.7289999999999992},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":8.2100000000000009}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":8.3049999999999997},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":8.5259999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":15.978999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345036,"kb_used_data":93072,"kb_used_omap":0,"kb_used_meta":251904,"kb_avail":94026804,"statfs":{"total":96636764160,"available":96283447296,"internally_reserved":0,"allocated":95305728,"data_stored":95032606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":257949696},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.391},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.83399999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":4.867},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":8.7870000000000008}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.92700000000000005}]}]},{"osd":0,"up_from":22,"seq":94489280630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":355056,"kb_used_data":96676,"kb_used_omap":0,"kb_used_meta":258367,"kb_avail":94016784,"statfs":{"total":96636764160,"available":96273186816,"internally_reserved":0,"allocated":98996224,"data_stored":98723102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264567860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.60499999999999998},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:47:58 2021","interfaces":[{"interface":"back","average":{"1min":5.2910000000000004,"5min":3.5099999999999998,"15min":2.8540000000000001},"min":{"1min":0.436,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":25.239999999999998,"5min":25.239999999999998,"15min":27.77},"last":4.9950000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.585,"15min":2.6560000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2769999999999992,"5min":12.413,"15min":22.765000000000001},"last":1.0369999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":9.6980000000000004},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":0.67400000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335460,"kb_used_data":96120,"kb_used_omap":0,"kb_used_meta":239295,"kb_avail":94036380,"statfs":{"total":96636764160,"available":96293253120,"internally_reserved":0,"allocated":98426880,"data_stored":98153758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":245038132},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.86299999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.748},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.97999999999999998}]}]}]}} 2021-01-15T18:49:06.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:08.000 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:08.000 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4396/91412 objects degraded (4.809%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":598,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22853,"data_bytes":187091244,"bytes_used":1408159744,"bytes_avail":385138896896,"bytes_total":386547056640,"degraded_objects":4396,"degraded_total":91412,"degraded_ratio":0.048089966306393034,"recovering_objects_per_sec":64,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":775,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":145},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2021-01-15T18:48:46.626754+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:49:08.119 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:08.120 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:09.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:09.392 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:09.429 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2021-01-15T18:49:08.639808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":187091244,"num_objects":22853,"num_object_clones":0,"num_object_copies":91412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131872,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17011,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131899,"ondisk_log_size":131899,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1375156,"kb_used_data":369260,"kb_used_omap":2,"kb_used_meta":1005693,"kb_avail":376112204,"statfs":{"total":386547056640,"available":385138896896,"internally_reserved":0,"allocated":378122240,"data_stored":377029752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1029829788},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,1,2,8,20,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14344192,"num_objects":-1751,"num_object_clones":0,"num_object_copies":-7004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1751,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010183"},"pg_stats":[{"pgid":"1.f","version":"52'8050","reported_seq":"8320","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:05.558327+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:05.558327+0000","last_peered":"2021-01-15T18:49:05.558327+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:05.558327+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8050,"ondisk_log_size":8050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11943936,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8050,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8076","reported_seq":"11012","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.446637+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:05.446637+0000","last_peered":"2021-01-15T18:49:05.446637+0000","last_clean":"2021-01-15T18:49:05.446637+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:05.446637+0000","last_undegraded":"2021-01-15T18:49:05.446637+0000","last_fullsized":"2021-01-15T18:49:05.446637+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8076,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11346086,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8086","reported_seq":"11027","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:04.099594+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:04.099594+0000","last_peered":"2021-01-15T18:49:04.099594+0000","last_clean":"2021-01-15T18:49:04.099594+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:04.099594+0000","last_undegraded":"2021-01-15T18:49:04.099594+0000","last_fullsized":"2021-01-15T18:49:04.099594+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8086,"ondisk_log_size":8086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11513963,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8085,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8362","reported_seq":"8616","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:05.557146+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:05.557146+0000","last_peered":"2021-01-15T18:49:05.557146+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:05.557146+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8362,"ondisk_log_size":8362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11714560,"num_objects":1430,"num_object_clones":0,"num_object_copies":5720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8362,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":96}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8408","reported_seq":"9012","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.615426+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:05.615426+0000","last_peered":"2021-01-15T18:49:05.615426+0000","last_clean":"2021-01-15T18:49:05.615426+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:05.615426+0000","last_undegraded":"2021-01-15T18:49:05.615426+0000","last_fullsized":"2021-01-15T18:49:05.615426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8408,"ondisk_log_size":8408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11632806,"num_objects":1421,"num_object_clones":0,"num_object_copies":5684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8406,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8140","reported_seq":"8363","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.739047+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:05.739047+0000","last_peered":"2021-01-15T18:49:05.739047+0000","last_clean":"2021-01-15T18:49:05.739047+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:05.739047+0000","last_undegraded":"2021-01-15T18:49:05.739047+0000","last_fullsized":"2021-01-15T18:49:05.739047+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8140,"ondisk_log_size":8140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11354278,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8138,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8243","reported_seq":"11116","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.851737+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:05.851737+0000","last_peered":"2021-01-15T18:49:05.851737+0000","last_clean":"2021-01-15T18:49:05.851737+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:05.851737+0000","last_undegraded":"2021-01-15T18:49:05.851737+0000","last_fullsized":"2021-01-15T18:49:05.851737+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8243,"ondisk_log_size":8243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12140710,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8384","reported_seq":"11393","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.819900+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:05.819900+0000","last_peered":"2021-01-15T18:49:05.819900+0000","last_clean":"2021-01-15T18:49:05.819900+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:05.819900+0000","last_undegraded":"2021-01-15T18:49:05.819900+0000","last_fullsized":"2021-01-15T18:49:05.819900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8384,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12062803,"num_objects":1473,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8383,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8337","reported_seq":"11284","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.863137+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:05.863137+0000","last_peered":"2021-01-15T18:49:05.863137+0000","last_clean":"2021-01-15T18:49:05.863137+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:05.863137+0000","last_undegraded":"2021-01-15T18:49:05.863137+0000","last_fullsized":"2021-01-15T18:49:05.863137+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8337,"ondisk_log_size":8337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11399251,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8336,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8314","reported_seq":"11267","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.429512+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:05.429512+0000","last_peered":"2021-01-15T18:49:05.429512+0000","last_clean":"2021-01-15T18:49:05.429512+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:05.429512+0000","last_undegraded":"2021-01-15T18:49:05.429512+0000","last_fullsized":"2021-01-15T18:49:05.429512+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8314,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11825401,"num_objects":1445,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8126","reported_seq":"10117","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:05.679450+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:05.679450+0000","last_peered":"2021-01-15T18:49:05.679450+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:05.679450+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8126,"ondisk_log_size":8126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11698342,"num_objects":1429,"num_object_clones":0,"num_object_copies":5716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8140","reported_seq":"10981","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.716556+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:05.716556+0000","last_peered":"2021-01-15T18:49:05.716556+0000","last_clean":"2021-01-15T18:49:05.716556+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:05.716556+0000","last_undegraded":"2021-01-15T18:49:05.716556+0000","last_fullsized":"2021-01-15T18:49:05.716556+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8140,"ondisk_log_size":8140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11690150,"num_objects":1428,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8138,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8064","reported_seq":"10968","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:04.649549+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:04.649549+0000","last_peered":"2021-01-15T18:49:04.649549+0000","last_clean":"2021-01-15T18:49:04.649549+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:04.649549+0000","last_undegraded":"2021-01-15T18:49:04.649549+0000","last_fullsized":"2021-01-15T18:49:04.649549+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8064,"ondisk_log_size":8064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11559078,"num_objects":1412,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8499","reported_seq":"8737","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:05.868227+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:05.868227+0000","last_peered":"2021-01-15T18:49:05.868227+0000","last_clean":"2021-01-15T18:49:05.868227+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:05.868227+0000","last_undegraded":"2021-01-15T18:49:05.868227+0000","last_fullsized":"2021-01-15T18:49:05.868227+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8499,"ondisk_log_size":8499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11362470,"num_objects":1388,"num_object_clones":0,"num_object_copies":5552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8497,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8572","reported_seq":"8850","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:05.516068+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:05.516068+0000","last_peered":"2021-01-15T18:49:05.516068+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:05.516068+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8572,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12042572,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8569,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":87}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8098","reported_seq":"10975","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:04.099146+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:04.099146+0000","last_peered":"2021-01-15T18:49:04.099146+0000","last_clean":"2021-01-15T18:49:04.099146+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:04.099146+0000","last_undegraded":"2021-01-15T18:49:04.099146+0000","last_fullsized":"2021-01-15T18:49:04.099146+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8098,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11804838,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8096,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":187091244,"num_objects":22853,"num_object_clones":0,"num_object_copies":91412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131872,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17011,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375480320,"data_stored":375480320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131899,"ondisk_log_size":131899,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348160,"kb_used_data":91760,"kb_used_omap":0,"kb_used_meta":256383,"kb_avail":94023680,"statfs":{"total":96636764160,"available":96280248320,"internally_reserved":0,"allocated":93962240,"data_stored":93689118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":262536244},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":9.5530000000000008},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":5.0810000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":21.515000000000001},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":21.617000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":7.8819999999999997},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.61499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345036,"kb_used_data":93072,"kb_used_omap":0,"kb_used_meta":251904,"kb_avail":94026804,"statfs":{"total":96636764160,"available":96283447296,"internally_reserved":0,"allocated":95305728,"data_stored":95032606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":257949696},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.391},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.83399999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":4.867},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":8.7870000000000008}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.92700000000000005}]}]},{"osd":0,"up_from":22,"seq":94489280631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340528,"kb_used_data":92520,"kb_used_omap":0,"kb_used_meta":247935,"kb_avail":94031312,"statfs":{"total":96636764160,"available":96288063488,"internally_reserved":0,"allocated":94740480,"data_stored":94467358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":253885492},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,1,1,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":6.1849999999999996},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":6.2750000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":11.773},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":11.590999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:04 2021","interfaces":[{"interface":"back","average":{"1min":4.5410000000000004,"5min":4.6200000000000001,"15min":4.4560000000000004},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.262,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":3.2730000000000001,"5min":4.7329999999999997,"15min":3.3879999999999999},"min":{"1min":0.64000000000000001,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":17.001999999999999,"5min":56.927,"15min":56.927},"last":10.679}]}]},{"osd":1,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341432,"kb_used_data":91908,"kb_used_omap":0,"kb_used_meta":249471,"kb_avail":94030408,"statfs":{"total":96636764160,"available":96287137792,"internally_reserved":0,"allocated":94113792,"data_stored":93840670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255458356},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.86299999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.748},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.97999999999999998}]}]}]}} 2021-01-15T18:49:12.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:13.193 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:13.240 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2021-01-15T18:49:12.641273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179079468,"num_objects":21875,"num_object_clones":0,"num_object_copies":87500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132850,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17060,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132877,"ondisk_log_size":132877,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1391196,"kb_used_data":353640,"kb_used_omap":2,"kb_used_meta":1037373,"kb_avail":376096164,"statfs":{"total":386547056640,"available":385122471936,"internally_reserved":0,"allocated":362127360,"data_stored":361034872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1062270108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,17,13,18,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":126,"apply_latency_ms":126,"commit_latency_ns":126000000,"apply_latency_ns":126000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21708800,"num_objects":-2650,"num_object_clones":0,"num_object_copies":-10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010891"},"pg_stats":[{"pgid":"1.f","version":"52'8095","reported_seq":"8364","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.655183+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:10.655183+0000","last_peered":"2021-01-15T18:49:10.655183+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:10.655183+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8095,"ondisk_log_size":8095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11575296,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8095,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8139","reported_seq":"11074","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.765493+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:10.765493+0000","last_peered":"2021-01-15T18:49:10.765493+0000","last_clean":"2021-01-15T18:49:10.765493+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:10.765493+0000","last_undegraded":"2021-01-15T18:49:10.765493+0000","last_fullsized":"2021-01-15T18:49:10.765493+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8139,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10829990,"num_objects":1323,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8141","reported_seq":"11083","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:09.132503+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:09.132503+0000","last_peered":"2021-01-15T18:49:09.132503+0000","last_clean":"2021-01-15T18:49:09.132503+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:09.132503+0000","last_undegraded":"2021-01-15T18:49:09.132503+0000","last_fullsized":"2021-01-15T18:49:09.132503+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8141,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11063403,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8140,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8422","reported_seq":"8680","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.502849+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:10.502849+0000","last_peered":"2021-01-15T18:49:10.502849+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:10.502849+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8422,"ondisk_log_size":8422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11223040,"num_objects":1370,"num_object_clones":0,"num_object_copies":5480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8457","reported_seq":"9061","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.706260+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:10.706260+0000","last_peered":"2021-01-15T18:49:10.706260+0000","last_clean":"2021-01-15T18:49:10.706260+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:10.706260+0000","last_undegraded":"2021-01-15T18:49:10.706260+0000","last_fullsized":"2021-01-15T18:49:10.706260+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8457,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11231398,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8195","reported_seq":"8418","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.764053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:10.764053+0000","last_peered":"2021-01-15T18:49:10.764053+0000","last_clean":"2021-01-15T18:49:10.764053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:10.764053+0000","last_undegraded":"2021-01-15T18:49:10.764053+0000","last_fullsized":"2021-01-15T18:49:10.764053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8195,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10903718,"num_objects":1332,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8303","reported_seq":"11176","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.584164+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:10.584164+0000","last_peered":"2021-01-15T18:49:10.584164+0000","last_clean":"2021-01-15T18:49:10.584164+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:10.584164+0000","last_undegraded":"2021-01-15T18:49:10.584164+0000","last_fullsized":"2021-01-15T18:49:10.584164+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8303,"ondisk_log_size":8303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11649190,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8443","reported_seq":"11452","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.654736+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:10.654736+0000","last_peered":"2021-01-15T18:49:10.654736+0000","last_clean":"2021-01-15T18:49:10.654736+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:10.654736+0000","last_undegraded":"2021-01-15T18:49:10.654736+0000","last_fullsized":"2021-01-15T18:49:10.654736+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8443,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11579475,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8442,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8392","reported_seq":"11338","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.765650+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:10.765650+0000","last_peered":"2021-01-15T18:49:10.765650+0000","last_clean":"2021-01-15T18:49:10.765650+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:10.765650+0000","last_undegraded":"2021-01-15T18:49:10.765650+0000","last_fullsized":"2021-01-15T18:49:10.765650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8392,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10948691,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8392","reported_seq":"11345","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.580878+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:10.580878+0000","last_peered":"2021-01-15T18:49:10.580878+0000","last_clean":"2021-01-15T18:49:10.580878+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:10.580878+0000","last_undegraded":"2021-01-15T18:49:10.580878+0000","last_fullsized":"2021-01-15T18:49:10.580878+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8392,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11186425,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8195","reported_seq":"10286","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.639552+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:10.639552+0000","last_peered":"2021-01-15T18:49:10.639552+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:10.639552+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8195,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11133094,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8198","reported_seq":"11039","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.847298+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:10.847298+0000","last_peered":"2021-01-15T18:49:10.847298+0000","last_clean":"2021-01-15T18:49:10.847298+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:10.847298+0000","last_undegraded":"2021-01-15T18:49:10.847298+0000","last_fullsized":"2021-01-15T18:49:10.847298+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8198,"ondisk_log_size":8198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11215014,"num_objects":1370,"num_object_clones":0,"num_object_copies":5480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8196,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8140","reported_seq":"11045","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:09.655206+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:09.655206+0000","last_peered":"2021-01-15T18:49:09.655206+0000","last_clean":"2021-01-15T18:49:09.655206+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:09.655206+0000","last_undegraded":"2021-01-15T18:49:09.655206+0000","last_fullsized":"2021-01-15T18:49:09.655206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8140,"ondisk_log_size":8140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10936486,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8138,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8569","reported_seq":"8807","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.794316+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:10.794316+0000","last_peered":"2021-01-15T18:49:10.794316+0000","last_clean":"2021-01-15T18:49:10.794316+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:10.794316+0000","last_undegraded":"2021-01-15T18:49:10.794316+0000","last_fullsized":"2021-01-15T18:49:10.794316+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8569,"ondisk_log_size":8569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10789030,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8567,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8625","reported_seq":"8903","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.621589+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:10.621589+0000","last_peered":"2021-01-15T18:49:10.621589+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:10.621589+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8625,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608396,"num_objects":1419,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8171","reported_seq":"11048","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:09.132327+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:09.132327+0000","last_peered":"2021-01-15T18:49:09.132327+0000","last_clean":"2021-01-15T18:49:09.132327+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:09.132327+0000","last_undegraded":"2021-01-15T18:49:09.132327+0000","last_fullsized":"2021-01-15T18:49:09.132327+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8171,"ondisk_log_size":8171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11206822,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":179079468,"num_objects":21875,"num_object_clones":0,"num_object_copies":87500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132850,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17060,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":359370752,"data_stored":359370752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132877,"ondisk_log_size":132877,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350140,"kb_used_data":87860,"kb_used_omap":0,"kb_used_meta":262207,"kb_avail":94021700,"statfs":{"total":96636764160,"available":96278220800,"internally_reserved":0,"allocated":89968640,"data_stored":89695518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":268500020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":2.536},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.624}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.9140000000000001},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":3.0179999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.1549999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346856,"kb_used_data":89024,"kb_used_omap":0,"kb_used_meta":257792,"kb_avail":94024984,"statfs":{"total":96636764160,"available":96281583616,"internally_reserved":0,"allocated":91160576,"data_stored":90887454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":263979008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,10,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.78700000000000003}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.1319999999999999},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.46300000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.3319999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":5.8310000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346564,"kb_used_data":88688,"kb_used_omap":0,"kb_used_meta":257855,"kb_avail":94025276,"statfs":{"total":96636764160,"available":96281882624,"internally_reserved":0,"allocated":90816512,"data_stored":90543390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264043572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.80300000000000005},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.103}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.177},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.88700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.6079999999999997},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.97699999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347636,"kb_used_data":88068,"kb_used_omap":0,"kb_used_meta":259519,"kb_avail":94024204,"statfs":{"total":96636764160,"available":96280784896,"internally_reserved":0,"allocated":90181632,"data_stored":89908510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":265747508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":6.0469999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":13.122},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.2799999999999998},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.3730000000000002}]}]}]}} 2021-01-15T18:49:13.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:14.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:14.722 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4347/87500 objects degraded (4.968%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":605,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21875,"data_bytes":179079468,"bytes_used":1424584704,"bytes_avail":385122471936,"bytes_total":386547056640,"degraded_objects":4347,"degraded_total":87500,"degraded_ratio":0.049680000000000002,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":292,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2021-01-15T18:48:46.626754+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2021-01-15T18:49:14.767 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:14.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:16.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:16.106 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:16.129 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2021-01-15T18:49:14.641702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":178096428,"num_objects":21755,"num_object_clones":0,"num_object_copies":87020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132970,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17060,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132997,"ondisk_log_size":132997,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1380876,"kb_used_data":349688,"kb_used_omap":2,"kb_used_meta":1030973,"kb_avail":376106484,"statfs":{"total":386547056640,"available":385133039616,"internally_reserved":0,"allocated":358080512,"data_stored":356988024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1055716508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,16,7,21,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":127,"apply_latency_ms":127,"commit_latency_ns":127000000,"apply_latency_ns":127000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17408000,"num_objects":-2125,"num_object_clones":0,"num_object_copies":-8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008263"},"pg_stats":[{"pgid":"1.f","version":"52'8095","reported_seq":"8364","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.655183+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:10.655183+0000","last_peered":"2021-01-15T18:49:10.655183+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:10.655183+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8095,"ondisk_log_size":8095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11575296,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8095,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8139","reported_seq":"11074","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.765493+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:10.765493+0000","last_peered":"2021-01-15T18:49:10.765493+0000","last_clean":"2021-01-15T18:49:10.765493+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:10.765493+0000","last_undegraded":"2021-01-15T18:49:10.765493+0000","last_fullsized":"2021-01-15T18:49:10.765493+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8139,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10829990,"num_objects":1323,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8201","reported_seq":"11144","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:14.112323+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:14.112323+0000","last_peered":"2021-01-15T18:49:14.112323+0000","last_clean":"2021-01-15T18:49:14.112323+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:14.112323+0000","last_undegraded":"2021-01-15T18:49:14.112323+0000","last_fullsized":"2021-01-15T18:49:14.112323+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8201,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10571883,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8200,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8422","reported_seq":"8680","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.502849+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:10.502849+0000","last_peered":"2021-01-15T18:49:10.502849+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:10.502849+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8422,"ondisk_log_size":8422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11223040,"num_objects":1370,"num_object_clones":0,"num_object_copies":5480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8457","reported_seq":"9061","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.706260+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:10.706260+0000","last_peered":"2021-01-15T18:49:10.706260+0000","last_clean":"2021-01-15T18:49:10.706260+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:10.706260+0000","last_undegraded":"2021-01-15T18:49:10.706260+0000","last_fullsized":"2021-01-15T18:49:10.706260+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8457,"ondisk_log_size":8457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11231398,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8195","reported_seq":"8418","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.764053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:10.764053+0000","last_peered":"2021-01-15T18:49:10.764053+0000","last_clean":"2021-01-15T18:49:10.764053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:10.764053+0000","last_undegraded":"2021-01-15T18:49:10.764053+0000","last_fullsized":"2021-01-15T18:49:10.764053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8195,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10903718,"num_objects":1332,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8303","reported_seq":"11176","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.584164+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:10.584164+0000","last_peered":"2021-01-15T18:49:10.584164+0000","last_clean":"2021-01-15T18:49:10.584164+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:10.584164+0000","last_undegraded":"2021-01-15T18:49:10.584164+0000","last_fullsized":"2021-01-15T18:49:10.584164+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8303,"ondisk_log_size":8303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11649190,"num_objects":1423,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8443","reported_seq":"11452","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.654736+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:10.654736+0000","last_peered":"2021-01-15T18:49:10.654736+0000","last_clean":"2021-01-15T18:49:10.654736+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:10.654736+0000","last_undegraded":"2021-01-15T18:49:10.654736+0000","last_fullsized":"2021-01-15T18:49:10.654736+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8443,"ondisk_log_size":8443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11579475,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8442,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8392","reported_seq":"11338","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.765650+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:10.765650+0000","last_peered":"2021-01-15T18:49:10.765650+0000","last_clean":"2021-01-15T18:49:10.765650+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:10.765650+0000","last_undegraded":"2021-01-15T18:49:10.765650+0000","last_fullsized":"2021-01-15T18:49:10.765650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8392,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10948691,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8392","reported_seq":"11345","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.580878+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:10.580878+0000","last_peered":"2021-01-15T18:49:10.580878+0000","last_clean":"2021-01-15T18:49:10.580878+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:10.580878+0000","last_undegraded":"2021-01-15T18:49:10.580878+0000","last_fullsized":"2021-01-15T18:49:10.580878+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8392,"ondisk_log_size":8392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11186425,"num_objects":1367,"num_object_clones":0,"num_object_copies":5468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8195","reported_seq":"10286","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.639552+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:10.639552+0000","last_peered":"2021-01-15T18:49:10.639552+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:10.639552+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8195,"ondisk_log_size":8195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11133094,"num_objects":1360,"num_object_clones":0,"num_object_copies":5440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":985}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8198","reported_seq":"11039","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.847298+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:10.847298+0000","last_peered":"2021-01-15T18:49:10.847298+0000","last_clean":"2021-01-15T18:49:10.847298+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:10.847298+0000","last_undegraded":"2021-01-15T18:49:10.847298+0000","last_fullsized":"2021-01-15T18:49:10.847298+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8198,"ondisk_log_size":8198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11215014,"num_objects":1370,"num_object_clones":0,"num_object_copies":5480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8196,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8140","reported_seq":"11045","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:09.655206+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:09.655206+0000","last_peered":"2021-01-15T18:49:09.655206+0000","last_clean":"2021-01-15T18:49:09.655206+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:09.655206+0000","last_undegraded":"2021-01-15T18:49:09.655206+0000","last_fullsized":"2021-01-15T18:49:09.655206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8140,"ondisk_log_size":8140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10936486,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8138,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8569","reported_seq":"8807","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:10.794316+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:10.794316+0000","last_peered":"2021-01-15T18:49:10.794316+0000","last_clean":"2021-01-15T18:49:10.794316+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:10.794316+0000","last_undegraded":"2021-01-15T18:49:10.794316+0000","last_fullsized":"2021-01-15T18:49:10.794316+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8569,"ondisk_log_size":8569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10789030,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8567,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8625","reported_seq":"8903","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:10.621589+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:10.621589+0000","last_peered":"2021-01-15T18:49:10.621589+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:10.621589+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8625,"ondisk_log_size":8625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608396,"num_objects":1419,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8231","reported_seq":"11108","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:14.050827+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:14.050827+0000","last_peered":"2021-01-15T18:49:14.050827+0000","last_clean":"2021-01-15T18:49:14.050827+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:14.050827+0000","last_undegraded":"2021-01-15T18:49:14.050827+0000","last_fullsized":"2021-01-15T18:49:14.050827+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8231,"ondisk_log_size":8231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10715302,"num_objects":1309,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":178096428,"num_objects":21755,"num_object_clones":0,"num_object_copies":87020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132970,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17060,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355381248,"data_stored":355381248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132997,"ondisk_log_size":132997,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350140,"kb_used_data":87860,"kb_used_omap":0,"kb_used_meta":262207,"kb_avail":94021700,"statfs":{"total":96636764160,"available":96278220800,"internally_reserved":0,"allocated":89968640,"data_stored":89695518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":268500020},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":4.1879999999999997,"5min":3.3820000000000001,"15min":3.2370000000000001},"min":{"1min":0.53100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":15.265000000000001,"5min":24.763000000000002,"15min":24.763000000000002},"last":2.536},{"interface":"front","average":{"1min":2.7999999999999998,"5min":2.7709999999999999,"15min":3.2069999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":18.241,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.624}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":2.9140000000000001},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":3.0179999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:10 2021","interfaces":[{"interface":"back","average":{"1min":7.7480000000000002,"5min":4.4569999999999999,"15min":3.02},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":42.475000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":4.4749999999999996,"5min":4.5129999999999999,"15min":3.4649999999999999},"min":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":24.103000000000002,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.1549999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336536,"kb_used_data":85072,"kb_used_omap":0,"kb_used_meta":251392,"kb_avail":94035304,"statfs":{"total":96636764160,"available":96292151296,"internally_reserved":0,"allocated":87113728,"data_stored":86840606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":257425408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.44600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":4.3730000000000002},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":5.7300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.88100000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346564,"kb_used_data":88688,"kb_used_omap":0,"kb_used_meta":257855,"kb_avail":94025276,"statfs":{"total":96636764160,"available":96281882624,"internally_reserved":0,"allocated":90816512,"data_stored":90543390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264043572},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.80300000000000005},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.103}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.177},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.88700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.6079999999999997},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.97699999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347636,"kb_used_data":88068,"kb_used_omap":0,"kb_used_meta":259519,"kb_avail":94024204,"statfs":{"total":96636764160,"available":96280784896,"internally_reserved":0,"allocated":90181632,"data_stored":89908510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":265747508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":6.0469999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":13.122},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":2.2799999999999998},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":2.3730000000000002}]}]}]}} 2021-01-15T18:49:19.131 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:20.236 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:20.275 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:20.293 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2021-01-15T18:49:18.644888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":171043116,"num_objects":20894,"num_object_clones":0,"num_object_copies":83576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133831,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17117,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133858,"ondisk_log_size":133858,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1345656,"kb_used_data":337628,"kb_used_omap":2,"kb_used_meta":1007869,"kb_avail":376141704,"statfs":{"total":386547056640,"available":385169104896,"internally_reserved":0,"allocated":345731072,"data_stored":344638584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1032058012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,8,4,11,31,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16048128,"num_objects":-1959,"num_object_clones":0,"num_object_copies":-7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010664"},"pg_stats":[{"pgid":"1.f","version":"52'8171","reported_seq":"8440","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:15.684686+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:15.684686+0000","last_peered":"2021-01-15T18:49:15.684686+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:15.684686+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8171,"ondisk_log_size":8171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10952704,"num_objects":1337,"num_object_clones":0,"num_object_copies":5348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8171,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":87}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8198","reported_seq":"11134","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.766068+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:15.766068+0000","last_peered":"2021-01-15T18:49:15.766068+0000","last_clean":"2021-01-15T18:49:15.766068+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:15.766068+0000","last_undegraded":"2021-01-15T18:49:15.766068+0000","last_fullsized":"2021-01-15T18:49:15.766068+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8198,"ondisk_log_size":8198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10346662,"num_objects":1264,"num_object_clones":0,"num_object_copies":5056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8196,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8201","reported_seq":"11144","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:14.112323+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:14.112323+0000","last_peered":"2021-01-15T18:49:14.112323+0000","last_clean":"2021-01-15T18:49:14.112323+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:14.112323+0000","last_undegraded":"2021-01-15T18:49:14.112323+0000","last_fullsized":"2021-01-15T18:49:14.112323+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8201,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10571883,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8200,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8490","reported_seq":"8748","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:15.588408+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:15.588408+0000","last_peered":"2021-01-15T18:49:15.588408+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:15.588408+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8490,"ondisk_log_size":8490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10665984,"num_objects":1302,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8518","reported_seq":"9122","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.840117+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:15.840117+0000","last_peered":"2021-01-15T18:49:15.840117+0000","last_clean":"2021-01-15T18:49:15.840117+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:15.840117+0000","last_undegraded":"2021-01-15T18:49:15.840117+0000","last_fullsized":"2021-01-15T18:49:15.840117+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8518,"ondisk_log_size":8518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10731686,"num_objects":1311,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8253","reported_seq":"8475","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.902956+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:15.902956+0000","last_peered":"2021-01-15T18:49:15.902956+0000","last_clean":"2021-01-15T18:49:15.902956+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:15.902956+0000","last_undegraded":"2021-01-15T18:49:15.902956+0000","last_fullsized":"2021-01-15T18:49:15.902956+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8253,"ondisk_log_size":8253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10428582,"num_objects":1274,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8355","reported_seq":"11228","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.762058+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:15.762058+0000","last_peered":"2021-01-15T18:49:15.762058+0000","last_clean":"2021-01-15T18:49:15.762058+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:15.762058+0000","last_undegraded":"2021-01-15T18:49:15.762058+0000","last_fullsized":"2021-01-15T18:49:15.762058+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8355,"ondisk_log_size":8355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11223206,"num_objects":1371,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8515","reported_seq":"11524","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.677833+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:15.677833+0000","last_peered":"2021-01-15T18:49:15.677833+0000","last_clean":"2021-01-15T18:49:15.677833+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:15.677833+0000","last_undegraded":"2021-01-15T18:49:15.677833+0000","last_fullsized":"2021-01-15T18:49:15.677833+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8515,"ondisk_log_size":8515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10989651,"num_objects":1342,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8456","reported_seq":"11403","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.807159+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:15.807159+0000","last_peered":"2021-01-15T18:49:15.807159+0000","last_clean":"2021-01-15T18:49:15.807159+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:15.807159+0000","last_undegraded":"2021-01-15T18:49:15.807159+0000","last_fullsized":"2021-01-15T18:49:15.807159+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8456,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10424403,"num_objects":1273,"num_object_clones":0,"num_object_copies":5092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8455,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8452","reported_seq":"11405","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.677563+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:15.677563+0000","last_peered":"2021-01-15T18:49:15.677563+0000","last_clean":"2021-01-15T18:49:15.677563+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:15.677563+0000","last_undegraded":"2021-01-15T18:49:15.677563+0000","last_fullsized":"2021-01-15T18:49:15.677563+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8452,"ondisk_log_size":8452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10694905,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8244","reported_seq":"10447","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:15.685063+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:15.685063+0000","last_peered":"2021-01-15T18:49:15.685063+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:15.685063+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8244,"ondisk_log_size":8244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10731686,"num_objects":1311,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8249","reported_seq":"11090","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.766920+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:15.766920+0000","last_peered":"2021-01-15T18:49:15.766920+0000","last_clean":"2021-01-15T18:49:15.766920+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:15.766920+0000","last_undegraded":"2021-01-15T18:49:15.766920+0000","last_fullsized":"2021-01-15T18:49:15.766920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8249,"ondisk_log_size":8249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10797222,"num_objects":1319,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8209","reported_seq":"11116","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:14.583348+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:14.583348+0000","last_peered":"2021-01-15T18:49:14.583348+0000","last_clean":"2021-01-15T18:49:14.583348+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:14.583348+0000","last_undegraded":"2021-01-15T18:49:14.583348+0000","last_fullsized":"2021-01-15T18:49:14.583348+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8209,"ondisk_log_size":8209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10371238,"num_objects":1267,"num_object_clones":0,"num_object_copies":5068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8626","reported_seq":"8864","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:15.761626+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:15.761626+0000","last_peered":"2021-01-15T18:49:15.761626+0000","last_clean":"2021-01-15T18:49:15.761626+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:15.761626+0000","last_undegraded":"2021-01-15T18:49:15.761626+0000","last_fullsized":"2021-01-15T18:49:15.761626+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8626,"ondisk_log_size":8626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10322086,"num_objects":1261,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8690","reported_seq":"8968","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:15.685369+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:15.685369+0000","last_peered":"2021-01-15T18:49:15.685369+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:15.685369+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8690,"ondisk_log_size":8690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11075916,"num_objects":1354,"num_object_clones":0,"num_object_copies":5416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8687,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-31}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8231","reported_seq":"11108","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:14.050827+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:14.050827+0000","last_peered":"2021-01-15T18:49:14.050827+0000","last_clean":"2021-01-15T18:49:14.050827+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:14.050827+0000","last_undegraded":"2021-01-15T18:49:14.050827+0000","last_fullsized":"2021-01-15T18:49:14.050827+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8231,"ondisk_log_size":8231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10715302,"num_objects":1309,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":171043116,"num_objects":20894,"num_object_clones":0,"num_object_copies":83576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133831,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17117,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343277568,"data_stored":343277568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133858,"ondisk_log_size":133858,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215221,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339724,"kb_used_data":83840,"kb_used_omap":0,"kb_used_meta":255871,"kb_avail":94032116,"statfs":{"total":96636764160,"available":96288886784,"internally_reserved":0,"allocated":85852160,"data_stored":85579038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":262011956},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":3.2450000000000001},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":3.3380000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.5980000000000001},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.028}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336536,"kb_used_data":85072,"kb_used_omap":0,"kb_used_meta":251392,"kb_avail":94035304,"statfs":{"total":96636764160,"available":96292151296,"internally_reserved":0,"allocated":87113728,"data_stored":86840606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":257425408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,4,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.44600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":4.3730000000000002},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":5.7300000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.88100000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336220,"kb_used_data":84648,"kb_used_omap":0,"kb_used_meta":251519,"kb_avail":94035620,"statfs":{"total":96636764160,"available":96292474880,"internally_reserved":0,"allocated":86679552,"data_stored":86406430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":257555508},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":8.8859999999999992},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":9.0259999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":1.292}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.85999999999999999},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.97999999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333176,"kb_used_data":84068,"kb_used_omap":0,"kb_used_meta":249087,"kb_avail":94038664,"statfs":{"total":96636764160,"available":96295591936,"internally_reserved":0,"allocated":86085632,"data_stored":85812510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":255065140},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":8.3000000000000007},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.97399999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.236},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":1.079}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.175}]}]}]}} 2021-01-15T18:49:20.295 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:21.652 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:21.652 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4290/82840 objects degraded (5.179%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":612,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20710,"data_bytes":169535788,"bytes_used":1390116864,"bytes_avail":385156939776,"bytes_total":386547056640,"degraded_objects":4290,"degraded_total":82840,"degraded_ratio":0.051786576533075807,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":178},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:21.721 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:21.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:22.815 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:22.817 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:22.828 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2021-01-15T18:49:22.645715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":162728236,"num_objects":19879,"num_object_clones":0,"num_object_copies":79516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134846,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17163,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134873,"ondisk_log_size":134873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1369572,"kb_used_data":321460,"kb_used_omap":2,"kb_used_meta":1047997,"kb_avail":376117788,"statfs":{"total":386547056640,"available":385144614912,"internally_reserved":0,"allocated":329175040,"data_stored":328082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1073149084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,9,26,25,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":124,"apply_latency_ms":124,"commit_latency_ns":124000000,"apply_latency_ns":124000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-2770,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004870"},"pg_stats":[{"pgid":"1.f","version":"52'8233","reported_seq":"8503","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:20.591430+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:20.591430+0000","last_peered":"2021-01-15T18:49:20.591430+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:20.591430+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8233,"ondisk_log_size":8233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10444800,"num_objects":1275,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8233,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":25}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8265","reported_seq":"11201","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.810449+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:20.810449+0000","last_peered":"2021-01-15T18:49:20.810449+0000","last_clean":"2021-01-15T18:49:20.810449+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:20.810449+0000","last_undegraded":"2021-01-15T18:49:20.810449+0000","last_fullsized":"2021-01-15T18:49:20.810449+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8265,"ondisk_log_size":8265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9797798,"num_objects":1197,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8259","reported_seq":"11202","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:19.025984+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:19.025984+0000","last_peered":"2021-01-15T18:49:19.025984+0000","last_clean":"2021-01-15T18:49:19.025984+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:19.025984+0000","last_undegraded":"2021-01-15T18:49:19.025984+0000","last_fullsized":"2021-01-15T18:49:19.025984+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8259,"ondisk_log_size":8259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10096747,"num_objects":1234,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8258,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8552","reported_seq":"8809","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:20.427798+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:20.427798+0000","last_peered":"2021-01-15T18:49:20.427798+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:20.427798+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8552,"ondisk_log_size":8552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10158080,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8552,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-94}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8581","reported_seq":"9185","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.860755+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:20.860755+0000","last_peered":"2021-01-15T18:49:20.860755+0000","last_clean":"2021-01-15T18:49:20.860755+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:20.860755+0000","last_undegraded":"2021-01-15T18:49:20.860755+0000","last_fullsized":"2021-01-15T18:49:20.860755+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8581,"ondisk_log_size":8581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10215590,"num_objects":1248,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8309","reported_seq":"8532","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.337962+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:20.337962+0000","last_peered":"2021-01-15T18:49:20.337962+0000","last_clean":"2021-01-15T18:49:20.337962+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:20.337962+0000","last_undegraded":"2021-01-15T18:49:20.337962+0000","last_fullsized":"2021-01-15T18:49:20.337962+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8309,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9969830,"num_objects":1218,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8433","reported_seq":"11302","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.853509+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:20.853509+0000","last_peered":"2021-01-15T18:49:20.853509+0000","last_clean":"2021-01-15T18:49:20.853509+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:20.853509+0000","last_undegraded":"2021-01-15T18:49:20.853509+0000","last_fullsized":"2021-01-15T18:49:20.853509+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8433,"ondisk_log_size":8433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10584230,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8431,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8573","reported_seq":"11582","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.741060+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:20.741060+0000","last_peered":"2021-01-15T18:49:20.741060+0000","last_clean":"2021-01-15T18:49:20.741060+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:20.741060+0000","last_undegraded":"2021-01-15T18:49:20.741060+0000","last_fullsized":"2021-01-15T18:49:20.741060+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8573,"ondisk_log_size":8573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10514515,"num_objects":1284,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8519","reported_seq":"11465","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.833064+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:20.833064+0000","last_peered":"2021-01-15T18:49:20.833064+0000","last_clean":"2021-01-15T18:49:20.833064+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:20.833064+0000","last_undegraded":"2021-01-15T18:49:20.833064+0000","last_fullsized":"2021-01-15T18:49:20.833064+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8519,"ondisk_log_size":8519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9908307,"num_objects":1210,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8505","reported_seq":"11458","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.648071+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:20.648071+0000","last_peered":"2021-01-15T18:49:20.648071+0000","last_clean":"2021-01-15T18:49:20.648071+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:20.648071+0000","last_undegraded":"2021-01-15T18:49:20.648071+0000","last_fullsized":"2021-01-15T18:49:20.648071+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8505,"ondisk_log_size":8505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10260729,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8310","reported_seq":"10607","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:20.697518+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:20.697518+0000","last_peered":"2021-01-15T18:49:20.697518+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:20.697518+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8310,"ondisk_log_size":8310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10191014,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":973}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8315","reported_seq":"11156","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.930203+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:20.930203+0000","last_peered":"2021-01-15T18:49:20.930203+0000","last_clean":"2021-01-15T18:49:20.930203+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:20.930203+0000","last_undegraded":"2021-01-15T18:49:20.930203+0000","last_fullsized":"2021-01-15T18:49:20.930203+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8315,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10256550,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8268","reported_seq":"11175","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:19.658848+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:19.658848+0000","last_peered":"2021-01-15T18:49:19.658848+0000","last_clean":"2021-01-15T18:49:19.658848+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:19.658848+0000","last_undegraded":"2021-01-15T18:49:19.658848+0000","last_fullsized":"2021-01-15T18:49:19.658848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8268,"ondisk_log_size":8268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9887910,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8266,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8692","reported_seq":"8930","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:20.844859+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:20.844859+0000","last_peered":"2021-01-15T18:49:20.844859+0000","last_clean":"2021-01-15T18:49:20.844859+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:20.844859+0000","last_undegraded":"2021-01-15T18:49:20.844859+0000","last_fullsized":"2021-01-15T18:49:20.844859+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8692,"ondisk_log_size":8692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9781414,"num_objects":1195,"num_object_clones":0,"num_object_copies":4780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8761","reported_seq":"9039","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:20.630821+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:20.630821+0000","last_peered":"2021-01-15T18:49:20.630821+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:20.630821+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8761,"ondisk_log_size":8761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10494284,"num_objects":1283,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8758,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-102}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8298","reported_seq":"11175","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:18.880256+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:18.880256+0000","last_peered":"2021-01-15T18:49:18.880256+0000","last_clean":"2021-01-15T18:49:18.880256+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:18.880256+0000","last_undegraded":"2021-01-15T18:49:18.880256+0000","last_fullsized":"2021-01-15T18:49:18.880256+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8298,"ondisk_log_size":8298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10166438,"num_objects":1242,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":162728236,"num_objects":19879,"num_object_clones":0,"num_object_copies":79516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134846,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17163,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":326680576,"data_stored":326680576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134873,"ondisk_log_size":134873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345704,"kb_used_data":79760,"kb_used_omap":0,"kb_used_meta":265919,"kb_avail":94026136,"statfs":{"total":96636764160,"available":96282763264,"internally_reserved":0,"allocated":81674240,"data_stored":81401118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":272301108},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,8,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":1.228},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":1.421},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":1.5069999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.327}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342444,"kb_used_data":80992,"kb_used_omap":0,"kb_used_meta":261440,"kb_avail":94029396,"statfs":{"total":96636764160,"available":96286101504,"internally_reserved":0,"allocated":82935808,"data_stored":82662686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":267714560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":5.2990000000000004},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.70599999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.8390000000000004},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.95099999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342192,"kb_used_data":80720,"kb_used_omap":0,"kb_used_meta":261439,"kb_avail":94029648,"statfs":{"total":96636764160,"available":96286359552,"internally_reserved":0,"allocated":82657280,"data_stored":82384158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":267713588},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.748},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.88200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":1.0600000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339232,"kb_used_data":79988,"kb_used_omap":0,"kb_used_meta":259199,"kb_avail":94032608,"statfs":{"total":96636764160,"available":96289390592,"internally_reserved":0,"allocated":81907712,"data_stored":81634590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":265419828},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.745},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":0.80400000000000005}]}]}]}} 2021-01-15T18:49:25.830 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:27.093 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:27.102 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:27.146 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2021-01-15T18:49:26.649818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154470700,"num_objects":18871,"num_object_clones":0,"num_object_copies":75484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135854,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17212,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135881,"ondisk_log_size":135881,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1320392,"kb_used_data":306204,"kb_used_omap":2,"kb_used_meta":1014077,"kb_avail":376166968,"statfs":{"total":386547056640,"available":385194975232,"internally_reserved":0,"allocated":313552896,"data_stored":312460408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1038415004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,11,18,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23625728,"num_objects":-2884,"num_object_clones":0,"num_object_copies":-11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2884,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008115"},"pg_stats":[{"pgid":"1.f","version":"52'8299","reported_seq":"8569","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.584913+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:25.584913+0000","last_peered":"2021-01-15T18:49:25.584913+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:25.584913+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8299,"ondisk_log_size":8299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9904128,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-41}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8318","reported_seq":"11252","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.856139+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:25.856139+0000","last_peered":"2021-01-15T18:49:25.856139+0000","last_clean":"2021-01-15T18:49:25.856139+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:25.856139+0000","last_undegraded":"2021-01-15T18:49:25.856139+0000","last_fullsized":"2021-01-15T18:49:25.856139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8318,"ondisk_log_size":8318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9363622,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8316,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8333","reported_seq":"11276","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:23.958074+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:23.958074+0000","last_peered":"2021-01-15T18:49:23.958074+0000","last_clean":"2021-01-15T18:49:23.958074+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:23.958074+0000","last_undegraded":"2021-01-15T18:49:23.958074+0000","last_fullsized":"2021-01-15T18:49:23.958074+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8333,"ondisk_log_size":8333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9490539,"num_objects":1160,"num_object_clones":0,"num_object_copies":4640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8332,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8624","reported_seq":"8882","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.657803+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:25.657803+0000","last_peered":"2021-01-15T18:49:25.657803+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:25.657803+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8624,"ondisk_log_size":8624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9568256,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8641","reported_seq":"9243","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.913718+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:25.913718+0000","last_peered":"2021-01-15T18:49:25.913718+0000","last_clean":"2021-01-15T18:49:25.913718+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:25.913718+0000","last_undegraded":"2021-01-15T18:49:25.913718+0000","last_fullsized":"2021-01-15T18:49:25.913718+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8641,"ondisk_log_size":8641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9724070,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8366","reported_seq":"8589","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.811328+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:25.811328+0000","last_peered":"2021-01-15T18:49:25.811328+0000","last_clean":"2021-01-15T18:49:25.811328+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:25.811328+0000","last_undegraded":"2021-01-15T18:49:25.811328+0000","last_fullsized":"2021-01-15T18:49:25.811328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8366,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9502886,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8500","reported_seq":"11373","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.870847+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:25.870847+0000","last_peered":"2021-01-15T18:49:25.870847+0000","last_clean":"2021-01-15T18:49:25.870847+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:25.870847+0000","last_undegraded":"2021-01-15T18:49:25.870847+0000","last_fullsized":"2021-01-15T18:49:25.870847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8500,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10035366,"num_objects":1226,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8649","reported_seq":"11658","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.892516+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:25.892516+0000","last_peered":"2021-01-15T18:49:25.892516+0000","last_clean":"2021-01-15T18:49:25.892516+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:25.892516+0000","last_undegraded":"2021-01-15T18:49:25.892516+0000","last_fullsized":"2021-01-15T18:49:25.892516+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8649,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9891923,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8582","reported_seq":"11529","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.754909+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:25.754909+0000","last_peered":"2021-01-15T18:49:25.754909+0000","last_clean":"2021-01-15T18:49:25.754909+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:25.754909+0000","last_undegraded":"2021-01-15T18:49:25.754909+0000","last_fullsized":"2021-01-15T18:49:25.754909+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8582,"ondisk_log_size":8582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9392211,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8575","reported_seq":"11528","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.581229+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:25.581229+0000","last_peered":"2021-01-15T18:49:25.581229+0000","last_clean":"2021-01-15T18:49:25.581229+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:25.581229+0000","last_undegraded":"2021-01-15T18:49:25.581229+0000","last_fullsized":"2021-01-15T18:49:25.581229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8575,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9687289,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8366","reported_seq":"10760","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.768050+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:25.768050+0000","last_peered":"2021-01-15T18:49:25.768050+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:25.768050+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8366,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9732262,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8369","reported_seq":"11210","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.901804+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:25.901804+0000","last_peered":"2021-01-15T18:49:25.901804+0000","last_clean":"2021-01-15T18:49:25.901804+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:25.901804+0000","last_undegraded":"2021-01-15T18:49:25.901804+0000","last_fullsized":"2021-01-15T18:49:25.901804+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8369,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9814182,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8326","reported_seq":"11233","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:24.587871+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:24.587871+0000","last_peered":"2021-01-15T18:49:24.587871+0000","last_clean":"2021-01-15T18:49:24.587871+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:24.587871+0000","last_undegraded":"2021-01-15T18:49:24.587871+0000","last_fullsized":"2021-01-15T18:49:24.587871+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8326,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9412774,"num_objects":1150,"num_object_clones":0,"num_object_copies":4600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8743","reported_seq":"8981","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.852134+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:25.852134+0000","last_peered":"2021-01-15T18:49:25.852134+0000","last_clean":"2021-01-15T18:49:25.852134+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:25.852134+0000","last_undegraded":"2021-01-15T18:49:25.852134+0000","last_fullsized":"2021-01-15T18:49:25.852134+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8743,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9363622,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8821","reported_seq":"9099","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.704206+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:25.704206+0000","last_peered":"2021-01-15T18:49:25.704206+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:25.704206+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8821,"ondisk_log_size":8821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10002764,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8369","reported_seq":"11246","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:24.146865+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:24.146865+0000","last_peered":"2021-01-15T18:49:24.146865+0000","last_clean":"2021-01-15T18:49:24.146865+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:24.146865+0000","last_undegraded":"2021-01-15T18:49:24.146865+0000","last_fullsized":"2021-01-15T18:49:24.146865+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8369,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9584806,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154470700,"num_objects":18871,"num_object_clones":0,"num_object_copies":75484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135854,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17212,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135881,"ondisk_log_size":135881,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331680,"kb_used_data":76500,"kb_used_omap":0,"kb_used_meta":255103,"kb_avail":94040160,"statfs":{"total":96636764160,"available":96297123840,"internally_reserved":0,"allocated":78336000,"data_stored":78062878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261225524},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":25.114999999999998},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":25.295000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.9460000000000002},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.52300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332124,"kb_used_data":77072,"kb_used_omap":0,"kb_used_meta":255040,"kb_avail":94039716,"statfs":{"total":96636764160,"available":96296669184,"internally_reserved":0,"allocated":78921728,"data_stored":78648606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":261160960},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":5.2990000000000004},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.70599999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.8390000000000004},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.95099999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331808,"kb_used_data":76680,"kb_used_omap":0,"kb_used_meta":255103,"kb_avail":94040032,"statfs":{"total":96636764160,"available":96296992768,"internally_reserved":0,"allocated":78520320,"data_stored":78247198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261225524},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":1.845},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.95}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":9.9659999999999993},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.617},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":1.034}]}]},{"osd":1,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324780,"kb_used_data":75952,"kb_used_omap":0,"kb_used_meta":248831,"kb_avail":94047060,"statfs":{"total":96636764160,"available":96304189440,"internally_reserved":0,"allocated":77774848,"data_stored":77501726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":254802996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":50.558999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":49.470999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.1399999999999999}]}]}]}} 2021-01-15T18:49:27.147 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:28.435 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:28.435 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4195/75484 objects degraded (5.557%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":618,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18871,"data_bytes":154470700,"bytes_used":1352081408,"bytes_avail":385194975232,"bytes_total":386547056640,"degraded_objects":4195,"degraded_total":75484,"degraded_ratio":0.05557469132531398,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":240},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:28.460 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:28.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:29.134 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:49:29.133+0000 7f43a6ffd700 0 --2- 172.21.15.148:0/4006640390 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f43a811ead0 0x7f43a8120ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-01-15T18:49:29.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:29.727 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2021-01-15T18:49:28.650275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154470700,"num_objects":18871,"num_object_clones":0,"num_object_copies":75484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135854,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17212,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135881,"ondisk_log_size":135881,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1320392,"kb_used_data":306204,"kb_used_omap":2,"kb_used_meta":1014077,"kb_avail":376166968,"statfs":{"total":386547056640,"available":385194975232,"internally_reserved":0,"allocated":313552896,"data_stored":312460408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1038415004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,11,18,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16572416,"num_objects":-2023,"num_object_clones":0,"num_object_copies":-8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007469"},"pg_stats":[{"pgid":"1.f","version":"52'8299","reported_seq":"8569","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.584913+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:25.584913+0000","last_peered":"2021-01-15T18:49:25.584913+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:25.584913+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8299,"ondisk_log_size":8299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9904128,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-41}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8318","reported_seq":"11252","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.856139+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:25.856139+0000","last_peered":"2021-01-15T18:49:25.856139+0000","last_clean":"2021-01-15T18:49:25.856139+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:25.856139+0000","last_undegraded":"2021-01-15T18:49:25.856139+0000","last_fullsized":"2021-01-15T18:49:25.856139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8318,"ondisk_log_size":8318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9363622,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8316,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8333","reported_seq":"11276","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:23.958074+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:23.958074+0000","last_peered":"2021-01-15T18:49:23.958074+0000","last_clean":"2021-01-15T18:49:23.958074+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:23.958074+0000","last_undegraded":"2021-01-15T18:49:23.958074+0000","last_fullsized":"2021-01-15T18:49:23.958074+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8333,"ondisk_log_size":8333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9490539,"num_objects":1160,"num_object_clones":0,"num_object_copies":4640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8332,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8624","reported_seq":"8882","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.657803+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:25.657803+0000","last_peered":"2021-01-15T18:49:25.657803+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:25.657803+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8624,"ondisk_log_size":8624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9568256,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8641","reported_seq":"9243","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.913718+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:25.913718+0000","last_peered":"2021-01-15T18:49:25.913718+0000","last_clean":"2021-01-15T18:49:25.913718+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:25.913718+0000","last_undegraded":"2021-01-15T18:49:25.913718+0000","last_fullsized":"2021-01-15T18:49:25.913718+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8641,"ondisk_log_size":8641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9724070,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8366","reported_seq":"8589","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.811328+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:25.811328+0000","last_peered":"2021-01-15T18:49:25.811328+0000","last_clean":"2021-01-15T18:49:25.811328+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:25.811328+0000","last_undegraded":"2021-01-15T18:49:25.811328+0000","last_fullsized":"2021-01-15T18:49:25.811328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8366,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9502886,"num_objects":1161,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8500","reported_seq":"11373","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.870847+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:25.870847+0000","last_peered":"2021-01-15T18:49:25.870847+0000","last_clean":"2021-01-15T18:49:25.870847+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:25.870847+0000","last_undegraded":"2021-01-15T18:49:25.870847+0000","last_fullsized":"2021-01-15T18:49:25.870847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8500,"ondisk_log_size":8500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10035366,"num_objects":1226,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8649","reported_seq":"11658","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.892516+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:25.892516+0000","last_peered":"2021-01-15T18:49:25.892516+0000","last_clean":"2021-01-15T18:49:25.892516+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:25.892516+0000","last_undegraded":"2021-01-15T18:49:25.892516+0000","last_fullsized":"2021-01-15T18:49:25.892516+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8649,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9891923,"num_objects":1208,"num_object_clones":0,"num_object_copies":4832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8582","reported_seq":"11529","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.754909+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:25.754909+0000","last_peered":"2021-01-15T18:49:25.754909+0000","last_clean":"2021-01-15T18:49:25.754909+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:25.754909+0000","last_undegraded":"2021-01-15T18:49:25.754909+0000","last_fullsized":"2021-01-15T18:49:25.754909+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8582,"ondisk_log_size":8582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9392211,"num_objects":1147,"num_object_clones":0,"num_object_copies":4588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8575","reported_seq":"11528","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.581229+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:25.581229+0000","last_peered":"2021-01-15T18:49:25.581229+0000","last_clean":"2021-01-15T18:49:25.581229+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:25.581229+0000","last_undegraded":"2021-01-15T18:49:25.581229+0000","last_fullsized":"2021-01-15T18:49:25.581229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8575,"ondisk_log_size":8575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9687289,"num_objects":1184,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8366","reported_seq":"10760","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.768050+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:25.768050+0000","last_peered":"2021-01-15T18:49:25.768050+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:25.768050+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8366,"ondisk_log_size":8366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9732262,"num_objects":1189,"num_object_clones":0,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8369","reported_seq":"11210","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.901804+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:25.901804+0000","last_peered":"2021-01-15T18:49:25.901804+0000","last_clean":"2021-01-15T18:49:25.901804+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:25.901804+0000","last_undegraded":"2021-01-15T18:49:25.901804+0000","last_fullsized":"2021-01-15T18:49:25.901804+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8369,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9814182,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8326","reported_seq":"11233","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:24.587871+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:24.587871+0000","last_peered":"2021-01-15T18:49:24.587871+0000","last_clean":"2021-01-15T18:49:24.587871+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:24.587871+0000","last_undegraded":"2021-01-15T18:49:24.587871+0000","last_fullsized":"2021-01-15T18:49:24.587871+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8326,"ondisk_log_size":8326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9412774,"num_objects":1150,"num_object_clones":0,"num_object_copies":4600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8743","reported_seq":"8981","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:25.852134+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:25.852134+0000","last_peered":"2021-01-15T18:49:25.852134+0000","last_clean":"2021-01-15T18:49:25.852134+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:25.852134+0000","last_undegraded":"2021-01-15T18:49:25.852134+0000","last_fullsized":"2021-01-15T18:49:25.852134+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8743,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9363622,"num_objects":1144,"num_object_clones":0,"num_object_copies":4576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8821","reported_seq":"9099","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:25.704206+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:25.704206+0000","last_peered":"2021-01-15T18:49:25.704206+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:25.704206+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8821,"ondisk_log_size":8821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10002764,"num_objects":1223,"num_object_clones":0,"num_object_copies":4892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8369","reported_seq":"11246","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:24.146865+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:24.146865+0000","last_peered":"2021-01-15T18:49:24.146865+0000","last_clean":"2021-01-15T18:49:24.146865+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:24.146865+0000","last_undegraded":"2021-01-15T18:49:24.146865+0000","last_fullsized":"2021-01-15T18:49:24.146865+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8369,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9584806,"num_objects":1171,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":154470700,"num_objects":18871,"num_object_clones":0,"num_object_copies":75484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135854,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17212,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135881,"ondisk_log_size":135881,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215223,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331680,"kb_used_data":76500,"kb_used_omap":0,"kb_used_meta":255103,"kb_avail":94040160,"statfs":{"total":96636764160,"available":96297123840,"internally_reserved":0,"allocated":78336000,"data_stored":78062878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261225524},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":25.114999999999998},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":25.295000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.9460000000000002},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.52300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332124,"kb_used_data":77072,"kb_used_omap":0,"kb_used_meta":255040,"kb_avail":94039716,"statfs":{"total":96636764160,"available":96296669184,"internally_reserved":0,"allocated":78921728,"data_stored":78648606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":261160960},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,6,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":5.2990000000000004},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.70599999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.8390000000000004},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.95099999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331808,"kb_used_data":76680,"kb_used_omap":0,"kb_used_meta":255103,"kb_avail":94040032,"statfs":{"total":96636764160,"available":96296992768,"internally_reserved":0,"allocated":78520320,"data_stored":78247198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":261225524},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":1.845},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.95}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":9.9659999999999993},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.617},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":1.034}]}]},{"osd":1,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324780,"kb_used_data":75952,"kb_used_omap":0,"kb_used_meta":248831,"kb_avail":94047060,"statfs":{"total":96636764160,"available":96304189440,"internally_reserved":0,"allocated":77774848,"data_stored":77501726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":254802996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":50.558999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":49.470999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.1399999999999999}]}]}]}} 2021-01-15T18:49:32.730 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:34.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:34.197 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:34.240 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2021-01-15T18:49:32.651078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146540844,"num_objects":17903,"num_object_clones":0,"num_object_copies":71612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136822,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17269,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136849,"ondisk_log_size":136849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1340076,"kb_used_data":289668,"kb_used_omap":2,"kb_used_meta":1050237,"kb_avail":376147284,"statfs":{"total":386547056640,"available":385174818816,"internally_reserved":0,"allocated":296620032,"data_stored":295527544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1075442844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,1,15,15,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":131,"apply_latency_ms":131,"commit_latency_ns":131000000,"apply_latency_ns":131000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22994944,"num_objects":-2807,"num_object_clones":0,"num_object_copies":-11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2807,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005792"},"pg_stats":[{"pgid":"1.f","version":"52'8356","reported_seq":"8624","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:30.843925+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:30.843925+0000","last_peered":"2021-01-15T18:49:30.843925+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:30.843925+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8356,"ondisk_log_size":8356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9437184,"num_objects":1152,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-98}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8371","reported_seq":"11307","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.803134+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:30.803134+0000","last_peered":"2021-01-15T18:49:30.803134+0000","last_clean":"2021-01-15T18:49:30.803134+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:30.803134+0000","last_undegraded":"2021-01-15T18:49:30.803134+0000","last_fullsized":"2021-01-15T18:49:30.803134+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8371,"ondisk_log_size":8371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8929446,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8369,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8399","reported_seq":"11342","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:28.933231+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:28.933231+0000","last_peered":"2021-01-15T18:49:28.933231+0000","last_clean":"2021-01-15T18:49:28.933231+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:28.933231+0000","last_undegraded":"2021-01-15T18:49:28.933231+0000","last_fullsized":"2021-01-15T18:49:28.933231+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8399,"ondisk_log_size":8399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8949867,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8398,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8681","reported_seq":"8939","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:30.659769+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:30.659769+0000","last_peered":"2021-01-15T18:49:30.659769+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:30.659769+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8681,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9101312,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8681,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8698","reported_seq":"9302","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.877505+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:30.877505+0000","last_peered":"2021-01-15T18:49:30.877505+0000","last_clean":"2021-01-15T18:49:30.877505+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:30.877505+0000","last_undegraded":"2021-01-15T18:49:30.877505+0000","last_fullsized":"2021-01-15T18:49:30.877505+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8698,"ondisk_log_size":8698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9257126,"num_objects":1131,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8429","reported_seq":"8652","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.860804+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:30.860804+0000","last_peered":"2021-01-15T18:49:30.860804+0000","last_clean":"2021-01-15T18:49:30.860804+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:30.860804+0000","last_undegraded":"2021-01-15T18:49:30.860804+0000","last_fullsized":"2021-01-15T18:49:30.860804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8429,"ondisk_log_size":8429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8986790,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8552","reported_seq":"11425","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.765114+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:30.765114+0000","last_peered":"2021-01-15T18:49:30.765114+0000","last_clean":"2021-01-15T18:49:30.765114+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:30.765114+0000","last_undegraded":"2021-01-15T18:49:30.765114+0000","last_fullsized":"2021-01-15T18:49:30.765114+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8552,"ondisk_log_size":8552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9609382,"num_objects":1174,"num_object_clones":0,"num_object_copies":4696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8550,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8711","reported_seq":"11720","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.708262+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:30.708262+0000","last_peered":"2021-01-15T18:49:30.708262+0000","last_clean":"2021-01-15T18:49:30.708262+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:30.708262+0000","last_undegraded":"2021-01-15T18:49:30.708262+0000","last_fullsized":"2021-01-15T18:49:30.708262+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8711,"ondisk_log_size":8711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9384019,"num_objects":1146,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8647","reported_seq":"11592","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.985661+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:30.985661+0000","last_peered":"2021-01-15T18:49:30.985661+0000","last_clean":"2021-01-15T18:49:30.985661+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:30.985661+0000","last_undegraded":"2021-01-15T18:49:30.985661+0000","last_fullsized":"2021-01-15T18:49:30.985661+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8647,"ondisk_log_size":8647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8859731,"num_objects":1082,"num_object_clones":0,"num_object_copies":4328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8646,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8634","reported_seq":"11587","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.754181+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:30.754181+0000","last_peered":"2021-01-15T18:49:30.754181+0000","last_clean":"2021-01-15T18:49:30.754181+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:30.754181+0000","last_undegraded":"2021-01-15T18:49:30.754181+0000","last_fullsized":"2021-01-15T18:49:30.754181+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8634,"ondisk_log_size":8634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9203961,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8430","reported_seq":"10937","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:30.833987+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:30.833987+0000","last_peered":"2021-01-15T18:49:30.833987+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:30.833987+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8430,"ondisk_log_size":8430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9207974,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8435","reported_seq":"11276","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.477808+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:30.477808+0000","last_peered":"2021-01-15T18:49:30.477808+0000","last_clean":"2021-01-15T18:49:30.477808+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:30.477808+0000","last_undegraded":"2021-01-15T18:49:30.477808+0000","last_fullsized":"2021-01-15T18:49:30.477808+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8435,"ondisk_log_size":8435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9273510,"num_objects":1133,"num_object_clones":0,"num_object_copies":4532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8433,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8385","reported_seq":"11292","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:29.549906+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:29.549906+0000","last_peered":"2021-01-15T18:49:29.549906+0000","last_clean":"2021-01-15T18:49:29.549906+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:29.549906+0000","last_undegraded":"2021-01-15T18:49:29.549906+0000","last_fullsized":"2021-01-15T18:49:29.549906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8385,"ondisk_log_size":8385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8929446,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8383,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8789","reported_seq":"9027","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:30.752884+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:30.752884+0000","last_peered":"2021-01-15T18:49:30.752884+0000","last_clean":"2021-01-15T18:49:30.752884+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:30.752884+0000","last_undegraded":"2021-01-15T18:49:30.752884+0000","last_fullsized":"2021-01-15T18:49:30.752884+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8789,"ondisk_log_size":8789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8986790,"num_objects":1098,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8893","reported_seq":"9171","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:30.658521+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:30.658521+0000","last_peered":"2021-01-15T18:49:30.658521+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:30.658521+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8893,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9412940,"num_objects":1151,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8439","reported_seq":"11316","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:29.124170+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:29.124170+0000","last_peered":"2021-01-15T18:49:29.124170+0000","last_clean":"2021-01-15T18:49:29.124170+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:29.124170+0000","last_undegraded":"2021-01-15T18:49:29.124170+0000","last_fullsized":"2021-01-15T18:49:29.124170+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8439,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9011366,"num_objects":1101,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":146540844,"num_objects":17903,"num_object_clones":0,"num_object_copies":71612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136822,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17269,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294354944,"data_stored":294354944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136849,"ondisk_log_size":136849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215224,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337284,"kb_used_data":71824,"kb_used_omap":0,"kb_used_meta":265407,"kb_avail":94034556,"statfs":{"total":96636764160,"available":96291385344,"internally_reserved":0,"allocated":73547776,"data_stored":73274654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":271776820},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,9,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":38.780999999999999},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":42.018999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":3.4369999999999998},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":3.5529999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.57299999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338136,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":264960,"kb_avail":94033704,"statfs":{"total":96636764160,"available":96290512896,"internally_reserved":0,"allocated":74883072,"data_stored":74609950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":271319040},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.587},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.0860000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.5609999999999999},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.98399999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.696},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":1.175}]}]},{"osd":0,"up_from":22,"seq":94489280636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333784,"kb_used_data":72796,"kb_used_omap":0,"kb_used_meta":260927,"kb_avail":94038056,"statfs":{"total":96636764160,"available":96294969344,"internally_reserved":0,"allocated":74543104,"data_stored":74269982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":267189300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":0.93700000000000006},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.0289999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":9.6790000000000003},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":21.614999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.7450000000000001},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.67700000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330872,"kb_used_data":71920,"kb_used_omap":0,"kb_used_meta":258943,"kb_avail":94040968,"statfs":{"total":96636764160,"available":96297951232,"internally_reserved":0,"allocated":73646080,"data_stored":73372958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":265157684},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,1,8,3,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":17.588999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.502}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":17.523},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":24.795000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":24.582999999999998},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":24.696000000000002}]}]}]}} 2021-01-15T18:49:34.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:35.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4138/71080 objects degraded (5.822%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":626,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17770,"data_bytes":145451308,"bytes_used":1374629888,"bytes_avail":385172426752,"bytes_total":386547056640,"degraded_objects":4138,"degraded_total":71080,"degraded_ratio":0.058216094541361854,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:35.695 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:35.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:37.034 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:37.071 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:37.093 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2021-01-15T18:49:36.651937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138037548,"num_objects":16865,"num_object_clones":0,"num_object_copies":67460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137860,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137887,"ondisk_log_size":137887,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1352560,"kb_used_data":274848,"kb_used_omap":2,"kb_used_meta":1077565,"kb_avail":376134800,"statfs":{"total":386547056640,"available":385162035200,"internally_reserved":0,"allocated":281444352,"data_stored":280351864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1103426716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,9,4,6,11,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23502848,"num_objects":-2869,"num_object_clones":0,"num_object_copies":-11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005785"},"pg_stats":[{"pgid":"1.f","version":"52'8419","reported_seq":"8689","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.705723+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:35.705723+0000","last_peered":"2021-01-15T18:49:35.705723+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:35.705723+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8419,"ondisk_log_size":8419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8921088,"num_objects":1089,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8440","reported_seq":"11376","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.965868+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:35.965868+0000","last_peered":"2021-01-15T18:49:35.965868+0000","last_clean":"2021-01-15T18:49:35.965868+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:35.965868+0000","last_undegraded":"2021-01-15T18:49:35.965868+0000","last_fullsized":"2021-01-15T18:49:35.965868+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8440,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8364198,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8458","reported_seq":"11400","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:34.067403+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:34.067403+0000","last_peered":"2021-01-15T18:49:34.067403+0000","last_clean":"2021-01-15T18:49:34.067403+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:34.067403+0000","last_undegraded":"2021-01-15T18:49:34.067403+0000","last_fullsized":"2021-01-15T18:49:34.067403+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8458,"ondisk_log_size":8458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8466539,"num_objects":1035,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8457,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8738","reported_seq":"8996","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.754922+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:35.754922+0000","last_peered":"2021-01-15T18:49:35.754922+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:35.754922+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8738,"ondisk_log_size":8738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8634368,"num_objects":1054,"num_object_clones":0,"num_object_copies":4216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8766","reported_seq":"9369","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.961189+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:35.961189+0000","last_peered":"2021-01-15T18:49:35.961189+0000","last_clean":"2021-01-15T18:49:35.961189+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:35.961189+0000","last_undegraded":"2021-01-15T18:49:35.961189+0000","last_fullsized":"2021-01-15T18:49:35.961189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8766,"ondisk_log_size":8766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8700070,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8764,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8489","reported_seq":"8712","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.782035+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:35.782035+0000","last_peered":"2021-01-15T18:49:35.782035+0000","last_clean":"2021-01-15T18:49:35.782035+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:35.782035+0000","last_undegraded":"2021-01-15T18:49:35.782035+0000","last_fullsized":"2021-01-15T18:49:35.782035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8489,"ondisk_log_size":8489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8495270,"num_objects":1038,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8632","reported_seq":"11505","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:36.066359+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:36.066359+0000","last_peered":"2021-01-15T18:49:36.066359+0000","last_clean":"2021-01-15T18:49:36.066359+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:36.066359+0000","last_undegraded":"2021-01-15T18:49:36.066359+0000","last_fullsized":"2021-01-15T18:49:36.066359+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8632,"ondisk_log_size":8632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8954022,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8770","reported_seq":"11779","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.930911+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:35.930911+0000","last_peered":"2021-01-15T18:49:35.930911+0000","last_clean":"2021-01-15T18:49:35.930911+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:35.930911+0000","last_undegraded":"2021-01-15T18:49:35.930911+0000","last_fullsized":"2021-01-15T18:49:35.930911+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8770,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8900691,"num_objects":1087,"num_object_clones":0,"num_object_copies":4348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8702","reported_seq":"11649","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.734971+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:35.734971+0000","last_peered":"2021-01-15T18:49:35.734971+0000","last_clean":"2021-01-15T18:49:35.734971+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:35.734971+0000","last_undegraded":"2021-01-15T18:49:35.734971+0000","last_fullsized":"2021-01-15T18:49:35.734971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8702,"ondisk_log_size":8702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8409171,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8696","reported_seq":"11647","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.811186+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:35.811186+0000","last_peered":"2021-01-15T18:49:35.811186+0000","last_clean":"2021-01-15T18:49:35.811186+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:35.811186+0000","last_undegraded":"2021-01-15T18:49:35.811186+0000","last_fullsized":"2021-01-15T18:49:35.811186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8696,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8696057,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8476","reported_seq":"11075","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.805846+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:35.805846+0000","last_peered":"2021-01-15T18:49:35.805846+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:35.805846+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8476,"ondisk_log_size":8476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8831142,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8474,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8504","reported_seq":"11345","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:36.049036+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:36.049036+0000","last_peered":"2021-01-15T18:49:36.049036+0000","last_clean":"2021-01-15T18:49:36.049036+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:36.049036+0000","last_undegraded":"2021-01-15T18:49:36.049036+0000","last_fullsized":"2021-01-15T18:49:36.049036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8504,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8708262,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8455","reported_seq":"11362","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:34.728923+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:34.728923+0000","last_peered":"2021-01-15T18:49:34.728923+0000","last_clean":"2021-01-15T18:49:34.728923+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:34.728923+0000","last_undegraded":"2021-01-15T18:49:34.728923+0000","last_fullsized":"2021-01-15T18:49:34.728923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8455,"ondisk_log_size":8455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8356006,"num_objects":1021,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8864","reported_seq":"9102","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.954085+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:35.954085+0000","last_peered":"2021-01-15T18:49:35.954085+0000","last_clean":"2021-01-15T18:49:35.954085+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:35.954085+0000","last_undegraded":"2021-01-15T18:49:35.954085+0000","last_fullsized":"2021-01-15T18:49:35.954085+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8864,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8372390,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8965","reported_seq":"9242","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.697682+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:35.697682+0000","last_peered":"2021-01-15T18:49:35.697682+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:35.697682+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8965,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8823116,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8513","reported_seq":"11390","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:34.132485+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:34.132485+0000","last_peered":"2021-01-15T18:49:34.132485+0000","last_clean":"2021-01-15T18:49:34.132485+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:34.132485+0000","last_undegraded":"2021-01-15T18:49:34.132485+0000","last_fullsized":"2021-01-15T18:49:34.132485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8513,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8405158,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":138037548,"num_objects":16865,"num_object_clones":0,"num_object_copies":67460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137860,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277741568,"data_stored":277741568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137887,"ondisk_log_size":137887,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339552,"kb_used_data":68472,"kb_used_omap":0,"kb_used_meta":271039,"kb_avail":94032288,"statfs":{"total":96636764160,"available":96289062912,"internally_reserved":0,"allocated":70115328,"data_stored":69842206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":277543988},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":20.093},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":20.209},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":38.700000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.8010000000000002},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.2000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340472,"kb_used_data":69852,"kb_used_omap":0,"kb_used_meta":270592,"kb_avail":94031368,"statfs":{"total":96636764160,"available":96288120832,"internally_reserved":0,"allocated":71528448,"data_stored":71255326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":277086208},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.81799999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:48:37 2021","interfaces":[{"interface":"back","average":{"1min":3.831,"5min":3.3479999999999999,"15min":2.7360000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":24.071000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":6.7869999999999999},{"interface":"front","average":{"1min":3.629,"5min":2.3220000000000001,"15min":2.1320000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":12.074999999999999,"5min":20.98,"15min":20.98},"last":0.505}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.73199999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339756,"kb_used_data":68672,"kb_used_omap":0,"kb_used_meta":271039,"kb_avail":94032084,"statfs":{"total":96636764160,"available":96288854016,"internally_reserved":0,"allocated":70320128,"data_stored":70047006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":277543988},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:48:38 2021","interfaces":[{"interface":"back","average":{"1min":5.8140000000000001,"5min":6.04,"15min":3.8730000000000002},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":32.619999999999997,"5min":50.948,"15min":50.948},"last":5.6749999999999998},{"interface":"front","average":{"1min":5.5039999999999996,"5min":4.9020000000000001,"15min":3.5129999999999999},"min":{"1min":0.47099999999999997,"5min":0.375,"15min":0.375},"max":{"1min":32.720999999999997,"5min":50.031999999999996,"15min":50.031999999999996},"last":6.2060000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":9.5839999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.61599999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332780,"kb_used_data":67852,"kb_used_omap":0,"kb_used_meta":264895,"kb_avail":94039060,"statfs":{"total":96636764160,"available":96295997440,"internally_reserved":0,"allocated":69480448,"data_stored":69207326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":271252532},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.3620000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.75},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.85699999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0600000000000001}]}]}]}} 2021-01-15T18:49:40.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:41.060 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:41.069 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:41.091 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2021-01-15T18:49:40.653230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":136448300,"num_objects":16671,"num_object_clones":0,"num_object_copies":66684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16671,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138054,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138081,"ondisk_log_size":138081,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1331260,"kb_used_data":266044,"kb_used_omap":2,"kb_used_meta":1065085,"kb_avail":376156100,"statfs":{"total":386547056640,"available":385183846400,"internally_reserved":0,"allocated":272429056,"data_stored":271336568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1090647196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,7,4,4,17,9,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":106,"apply_latency_ms":106,"commit_latency_ns":106000000,"apply_latency_ns":106000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18022400,"num_objects":-2200,"num_object_clones":0,"num_object_copies":-8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002954"},"pg_stats":[{"pgid":"1.f","version":"52'8419","reported_seq":"8689","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.705723+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:35.705723+0000","last_peered":"2021-01-15T18:49:35.705723+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:35.705723+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8419,"ondisk_log_size":8419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8921088,"num_objects":1089,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8440","reported_seq":"11376","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.965868+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:35.965868+0000","last_peered":"2021-01-15T18:49:35.965868+0000","last_clean":"2021-01-15T18:49:35.965868+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:35.965868+0000","last_undegraded":"2021-01-15T18:49:35.965868+0000","last_fullsized":"2021-01-15T18:49:35.965868+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8440,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8364198,"num_objects":1022,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8512","reported_seq":"11455","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.154897+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:39.154897+0000","last_peered":"2021-01-15T18:49:39.154897+0000","last_clean":"2021-01-15T18:49:39.154897+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:39.154897+0000","last_undegraded":"2021-01-15T18:49:39.154897+0000","last_fullsized":"2021-01-15T18:49:39.154897+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8512,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8024171,"num_objects":981,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8511,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8738","reported_seq":"8996","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.754922+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:35.754922+0000","last_peered":"2021-01-15T18:49:35.754922+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:35.754922+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8738,"ondisk_log_size":8738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8634368,"num_objects":1054,"num_object_clones":0,"num_object_copies":4216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8766","reported_seq":"9369","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.961189+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:35.961189+0000","last_peered":"2021-01-15T18:49:35.961189+0000","last_clean":"2021-01-15T18:49:35.961189+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:35.961189+0000","last_undegraded":"2021-01-15T18:49:35.961189+0000","last_fullsized":"2021-01-15T18:49:35.961189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8766,"ondisk_log_size":8766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8700070,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8764,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8489","reported_seq":"8712","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.782035+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:35.782035+0000","last_peered":"2021-01-15T18:49:35.782035+0000","last_clean":"2021-01-15T18:49:35.782035+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:35.782035+0000","last_undegraded":"2021-01-15T18:49:35.782035+0000","last_fullsized":"2021-01-15T18:49:35.782035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8489,"ondisk_log_size":8489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8495270,"num_objects":1038,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8632","reported_seq":"11505","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:36.066359+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:36.066359+0000","last_peered":"2021-01-15T18:49:36.066359+0000","last_clean":"2021-01-15T18:49:36.066359+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:36.066359+0000","last_undegraded":"2021-01-15T18:49:36.066359+0000","last_fullsized":"2021-01-15T18:49:36.066359+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8632,"ondisk_log_size":8632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8954022,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8770","reported_seq":"11779","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.930911+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:35.930911+0000","last_peered":"2021-01-15T18:49:35.930911+0000","last_clean":"2021-01-15T18:49:35.930911+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:35.930911+0000","last_undegraded":"2021-01-15T18:49:35.930911+0000","last_fullsized":"2021-01-15T18:49:35.930911+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8770,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8900691,"num_objects":1087,"num_object_clones":0,"num_object_copies":4348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8702","reported_seq":"11649","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.734971+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:35.734971+0000","last_peered":"2021-01-15T18:49:35.734971+0000","last_clean":"2021-01-15T18:49:35.734971+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:35.734971+0000","last_undegraded":"2021-01-15T18:49:35.734971+0000","last_fullsized":"2021-01-15T18:49:35.734971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8702,"ondisk_log_size":8702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8409171,"num_objects":1027,"num_object_clones":0,"num_object_copies":4108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8701,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8696","reported_seq":"11647","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.811186+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:35.811186+0000","last_peered":"2021-01-15T18:49:35.811186+0000","last_clean":"2021-01-15T18:49:35.811186+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:35.811186+0000","last_undegraded":"2021-01-15T18:49:35.811186+0000","last_fullsized":"2021-01-15T18:49:35.811186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8696,"ondisk_log_size":8696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8696057,"num_objects":1063,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8476","reported_seq":"11075","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.805846+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:35.805846+0000","last_peered":"2021-01-15T18:49:35.805846+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:35.805846+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8476,"ondisk_log_size":8476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8831142,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8474,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8504","reported_seq":"11345","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:36.049036+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:36.049036+0000","last_peered":"2021-01-15T18:49:36.049036+0000","last_clean":"2021-01-15T18:49:36.049036+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:36.049036+0000","last_undegraded":"2021-01-15T18:49:36.049036+0000","last_fullsized":"2021-01-15T18:49:36.049036+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8504,"ondisk_log_size":8504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8708262,"num_objects":1064,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8502,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8522","reported_seq":"11428","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.816880+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:39.816880+0000","last_peered":"2021-01-15T18:49:39.816880+0000","last_clean":"2021-01-15T18:49:39.816880+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:39.816880+0000","last_undegraded":"2021-01-15T18:49:39.816880+0000","last_fullsized":"2021-01-15T18:49:39.816880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8522,"ondisk_log_size":8522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7807142,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8864","reported_seq":"9102","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:35.954085+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:35.954085+0000","last_peered":"2021-01-15T18:49:35.954085+0000","last_clean":"2021-01-15T18:49:35.954085+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:35.954085+0000","last_undegraded":"2021-01-15T18:49:35.954085+0000","last_fullsized":"2021-01-15T18:49:35.954085+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8864,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8372390,"num_objects":1023,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'8965","reported_seq":"9242","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:35.697682+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:35.697682+0000","last_peered":"2021-01-15T18:49:35.697682+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:35.697682+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8965,"ondisk_log_size":8965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8823116,"num_objects":1079,"num_object_clones":0,"num_object_copies":4316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8586","reported_seq":"11463","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.155809+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:39.155809+0000","last_peered":"2021-01-15T18:49:39.155809+0000","last_clean":"2021-01-15T18:49:39.155809+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:39.155809+0000","last_undegraded":"2021-01-15T18:49:39.155809+0000","last_fullsized":"2021-01-15T18:49:39.155809+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8586,"ondisk_log_size":8586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7807142,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":136448300,"num_objects":16671,"num_object_clones":0,"num_object_copies":66684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16671,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138054,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269496320,"data_stored":269496320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138081,"ondisk_log_size":138081,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339552,"kb_used_data":68472,"kb_used_omap":0,"kb_used_meta":271039,"kb_avail":94032288,"statfs":{"total":96636764160,"available":96289062912,"internally_reserved":0,"allocated":70115328,"data_stored":69842206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":277543988},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":20.093},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":20.209},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":38.700000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.8010000000000002},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.2000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329636,"kb_used_data":65116,"kb_used_omap":0,"kb_used_meta":264512,"kb_avail":94042204,"statfs":{"total":96636764160,"available":96299216896,"internally_reserved":0,"allocated":66678784,"data_stored":66405662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":270860288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":10.471},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":4.8109999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":5.6630000000000003},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":17.071000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":5.524},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":4.9139999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329292,"kb_used_data":64604,"kb_used_omap":0,"kb_used_meta":264639,"kb_avail":94042548,"statfs":{"total":96636764160,"available":96299569152,"internally_reserved":0,"allocated":66154496,"data_stored":65881374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270990388},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.88800000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.97799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.61599999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332780,"kb_used_data":67852,"kb_used_omap":0,"kb_used_meta":264895,"kb_avail":94039060,"statfs":{"total":96636764160,"available":96295997440,"internally_reserved":0,"allocated":69480448,"data_stored":69207326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":271252532},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.3620000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.75},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.85699999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0600000000000001}]}]}]}} 2021-01-15T18:49:41.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:42.810 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:42.811 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4093/66684 objects degraded (6.138%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":633,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":52,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16671,"data_bytes":136448300,"bytes_used":1363210240,"bytes_avail":385183846400,"bytes_total":386547056640,"degraded_objects":4093,"degraded_total":66684,"degraded_ratio":0.061379041449223203,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:42.921 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:42.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:44.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:44.149 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2021-01-15T18:49:42.653641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129960236,"num_objects":15879,"num_object_clones":0,"num_object_copies":63516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138846,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17361,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138873,"ondisk_log_size":138873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1310520,"kb_used_data":257948,"kb_used_omap":2,"kb_used_meta":1052413,"kb_avail":376176840,"statfs":{"total":386547056640,"available":385205084160,"internally_reserved":0,"allocated":264138752,"data_stored":263046264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1077671068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,3,11,16,17,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":103,"apply_latency_ms":103,"commit_latency_ns":103000000,"apply_latency_ns":103000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22913024,"num_objects":-2797,"num_object_clones":0,"num_object_copies":-11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2797,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002963"},"pg_stats":[{"pgid":"1.f","version":"52'8480","reported_seq":"8750","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:40.824002+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:40.824002+0000","last_peered":"2021-01-15T18:49:40.824002+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:40.824002+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8480,"ondisk_log_size":8480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8421376,"num_objects":1028,"num_object_clones":0,"num_object_copies":4112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8486","reported_seq":"11422","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.977190+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:40.977190+0000","last_peered":"2021-01-15T18:49:40.977190+0000","last_clean":"2021-01-15T18:49:40.977190+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:40.977190+0000","last_undegraded":"2021-01-15T18:49:40.977190+0000","last_fullsized":"2021-01-15T18:49:40.977190+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8486,"ondisk_log_size":8486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7987366,"num_objects":976,"num_object_clones":0,"num_object_copies":3904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8484,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8512","reported_seq":"11455","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.154897+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:39.154897+0000","last_peered":"2021-01-15T18:49:39.154897+0000","last_clean":"2021-01-15T18:49:39.154897+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:39.154897+0000","last_undegraded":"2021-01-15T18:49:39.154897+0000","last_fullsized":"2021-01-15T18:49:39.154897+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8512,"ondisk_log_size":8512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8024171,"num_objects":981,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8511,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8796","reported_seq":"9054","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:40.665803+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:40.665803+0000","last_peered":"2021-01-15T18:49:40.665803+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:40.665803+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8796,"ondisk_log_size":8796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8159232,"num_objects":996,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8832","reported_seq":"9433","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.923034+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:40.923034+0000","last_peered":"2021-01-15T18:49:40.923034+0000","last_clean":"2021-01-15T18:49:40.923034+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:40.923034+0000","last_undegraded":"2021-01-15T18:49:40.923034+0000","last_fullsized":"2021-01-15T18:49:40.923034+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8832,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8159398,"num_objects":997,"num_object_clones":0,"num_object_copies":3988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8543","reported_seq":"8766","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.944372+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:40.944372+0000","last_peered":"2021-01-15T18:49:40.944372+0000","last_clean":"2021-01-15T18:49:40.944372+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:40.944372+0000","last_undegraded":"2021-01-15T18:49:40.944372+0000","last_fullsized":"2021-01-15T18:49:40.944372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8543,"ondisk_log_size":8543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8052902,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8541,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8694","reported_seq":"11567","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.882242+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:40.882242+0000","last_peered":"2021-01-15T18:49:40.882242+0000","last_clean":"2021-01-15T18:49:40.882242+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:40.882242+0000","last_undegraded":"2021-01-15T18:49:40.882242+0000","last_fullsized":"2021-01-15T18:49:40.882242+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8694,"ondisk_log_size":8694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8446118,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8836","reported_seq":"11844","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:41.096274+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:41.096274+0000","last_peered":"2021-01-15T18:49:41.096274+0000","last_clean":"2021-01-15T18:49:41.096274+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:41.096274+0000","last_undegraded":"2021-01-15T18:49:41.096274+0000","last_fullsized":"2021-01-15T18:49:41.096274+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8836,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8360019,"num_objects":1021,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8757","reported_seq":"11704","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:41.052483+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:41.052483+0000","last_peered":"2021-01-15T18:49:41.052483+0000","last_clean":"2021-01-15T18:49:41.052483+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:41.052483+0000","last_undegraded":"2021-01-15T18:49:41.052483+0000","last_fullsized":"2021-01-15T18:49:41.052483+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8757,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7958611,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8757","reported_seq":"11710","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.517300+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:40.517300+0000","last_peered":"2021-01-15T18:49:40.517300+0000","last_clean":"2021-01-15T18:49:40.517300+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:40.517300+0000","last_undegraded":"2021-01-15T18:49:40.517300+0000","last_fullsized":"2021-01-15T18:49:40.517300+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8757,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8196345,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8536","reported_seq":"11229","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:40.824190+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:40.824190+0000","last_peered":"2021-01-15T18:49:40.824190+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:40.824190+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8536,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8339622,"num_objects":1019,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8572","reported_seq":"11412","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:40.977437+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:40.977437+0000","last_peered":"2021-01-15T18:49:40.977437+0000","last_clean":"2021-01-15T18:49:40.977437+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:40.977437+0000","last_undegraded":"2021-01-15T18:49:40.977437+0000","last_fullsized":"2021-01-15T18:49:40.977437+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8572,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8151206,"num_objects":996,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8522","reported_seq":"11428","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.816880+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:39.816880+0000","last_peered":"2021-01-15T18:49:39.816880+0000","last_clean":"2021-01-15T18:49:39.816880+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:39.816880+0000","last_undegraded":"2021-01-15T18:49:39.816880+0000","last_fullsized":"2021-01-15T18:49:39.816880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8522,"ondisk_log_size":8522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7807142,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8934","reported_seq":"9171","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:41.056139+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:41.056139+0000","last_peered":"2021-01-15T18:49:41.056139+0000","last_clean":"2021-01-15T18:49:41.056139+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:41.056139+0000","last_undegraded":"2021-01-15T18:49:41.056139+0000","last_fullsized":"2021-01-15T18:49:41.056139+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8934,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7798950,"num_objects":953,"num_object_clones":0,"num_object_copies":3812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'9030","reported_seq":"9308","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:40.842748+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:40.842748+0000","last_peered":"2021-01-15T18:49:40.842748+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:40.842748+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9030,"ondisk_log_size":9030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8290636,"num_objects":1014,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-371}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8586","reported_seq":"11463","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:39.155809+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:39.155809+0000","last_peered":"2021-01-15T18:49:39.155809+0000","last_clean":"2021-01-15T18:49:39.155809+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:39.155809+0000","last_undegraded":"2021-01-15T18:49:39.155809+0000","last_fullsized":"2021-01-15T18:49:39.155809+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8586,"ondisk_log_size":8586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7807142,"num_objects":954,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129960236,"num_objects":15879,"num_object_clones":0,"num_object_copies":63516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138846,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17361,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261324800,"data_stored":261324800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138873,"ondisk_log_size":138873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329112,"kb_used_data":64440,"kb_used_omap":0,"kb_used_meta":264639,"kb_avail":94042728,"statfs":{"total":96636764160,"available":96299753472,"internally_reserved":0,"allocated":65986560,"data_stored":65713438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270990388},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":2.5169999999999999},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.498}]},{"osd":1,"last update":"Fri Jan 15 18:48:43 2021","interfaces":[{"interface":"back","average":{"1min":5.6520000000000001,"5min":4.1699999999999999,"15min":3.6509999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":27.218,"5min":30.129000000000001,"15min":30.129000000000001},"last":11.214},{"interface":"front","average":{"1min":5.8230000000000004,"5min":4.3639999999999999,"15min":3.0379999999999998},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":28.155000000000001,"5min":28.155000000000001,"15min":28.155000000000001},"last":2.6259999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":10.288},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":2.7149999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329636,"kb_used_data":65116,"kb_used_omap":0,"kb_used_meta":264512,"kb_avail":94042204,"statfs":{"total":96636764160,"available":96299216896,"internally_reserved":0,"allocated":66678784,"data_stored":66405662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":270860288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":10.471},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":4.8109999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":5.6630000000000003},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":17.071000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":5.524},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":4.9139999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329292,"kb_used_data":64604,"kb_used_omap":0,"kb_used_meta":264639,"kb_avail":94042548,"statfs":{"total":96636764160,"available":96299569152,"internally_reserved":0,"allocated":66154496,"data_stored":65881374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270990388},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.88800000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.97799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.61599999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322480,"kb_used_data":63788,"kb_used_omap":0,"kb_used_meta":258623,"kb_avail":94049360,"statfs":{"total":96636764160,"available":96306544640,"internally_reserved":0,"allocated":65318912,"data_stored":65045790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264830004},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":1.782},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":0.749}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.81100000000000005},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.93300000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.0389999999999999}]}]}]}} 2021-01-15T18:49:47.153 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:48.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:48.432 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:48.449 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2021-01-15T18:49:46.654550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121882924,"num_objects":14893,"num_object_clones":0,"num_object_copies":59572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139832,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17411,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139859,"ondisk_log_size":139859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1326272,"kb_used_data":241624,"kb_used_omap":2,"kb_used_meta":1084477,"kb_avail":376161088,"statfs":{"total":386547056640,"available":385188954112,"internally_reserved":0,"allocated":247422976,"data_stored":246330488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1110504604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,6,13,20,23,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23568384,"num_objects":-2877,"num_object_clones":0,"num_object_copies":-11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003036"},"pg_stats":[{"pgid":"1.f","version":"52'8540","reported_seq":"8810","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.665983+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:45.665983+0000","last_peered":"2021-01-15T18:49:45.665983+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:45.665983+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8540,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7929856,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8532","reported_seq":"11468","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.014773+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:46.014773+0000","last_peered":"2021-01-15T18:49:46.014773+0000","last_clean":"2021-01-15T18:49:46.014773+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:46.014773+0000","last_undegraded":"2021-01-15T18:49:46.014773+0000","last_fullsized":"2021-01-15T18:49:46.014773+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8532,"ondisk_log_size":8532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7610534,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"52'8572","reported_seq":"11515","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:44.216346+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:44.216346+0000","last_peered":"2021-01-15T18:49:44.216346+0000","last_clean":"2021-01-15T18:49:44.216346+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:44.216346+0000","last_undegraded":"2021-01-15T18:49:44.216346+0000","last_fullsized":"2021-01-15T18:49:44.216346+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8572,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7532651,"num_objects":921,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8571,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8858","reported_seq":"9116","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.859748+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:45.859748+0000","last_peered":"2021-01-15T18:49:45.859748+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:45.859748+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8858,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7651328,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8892","reported_seq":"9496","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.740908+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:45.740908+0000","last_peered":"2021-01-15T18:49:45.740908+0000","last_clean":"2021-01-15T18:49:45.740908+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:45.740908+0000","last_undegraded":"2021-01-15T18:49:45.740908+0000","last_fullsized":"2021-01-15T18:49:45.740908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8892,"ondisk_log_size":8892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7667878,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8597","reported_seq":"8820","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.080611+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:46.080611+0000","last_peered":"2021-01-15T18:49:46.080611+0000","last_clean":"2021-01-15T18:49:46.080611+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:46.080611+0000","last_undegraded":"2021-01-15T18:49:46.080611+0000","last_fullsized":"2021-01-15T18:49:46.080611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8597,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7610534,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8748","reported_seq":"11620","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.974355+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:45.974355+0000","last_peered":"2021-01-15T18:49:45.974355+0000","last_clean":"2021-01-15T18:49:45.974355+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:45.974355+0000","last_undegraded":"2021-01-15T18:49:45.974355+0000","last_fullsized":"2021-01-15T18:49:45.974355+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8748,"ondisk_log_size":8748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8003750,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8900","reported_seq":"11909","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.771423+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:45.771423+0000","last_peered":"2021-01-15T18:49:45.771423+0000","last_clean":"2021-01-15T18:49:45.771423+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:45.771423+0000","last_undegraded":"2021-01-15T18:49:45.771423+0000","last_fullsized":"2021-01-15T18:49:45.771423+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8900,"ondisk_log_size":8900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7835731,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8830","reported_seq":"11777","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.015403+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:46.015403+0000","last_peered":"2021-01-15T18:49:46.015403+0000","last_clean":"2021-01-15T18:49:46.015403+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:46.015403+0000","last_undegraded":"2021-01-15T18:49:46.015403+0000","last_fullsized":"2021-01-15T18:49:46.015403+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7360595,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8830","reported_seq":"11783","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.862165+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:45.862165+0000","last_peered":"2021-01-15T18:49:45.862165+0000","last_clean":"2021-01-15T18:49:45.862165+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:45.862165+0000","last_undegraded":"2021-01-15T18:49:45.862165+0000","last_fullsized":"2021-01-15T18:49:45.862165+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7598329,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8602","reported_seq":"11393","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.857968+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:45.857968+0000","last_peered":"2021-01-15T18:49:45.857968+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:45.857968+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8602,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7798950,"num_objects":953,"num_object_clones":0,"num_object_copies":3812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8632","reported_seq":"11473","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.773170+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:45.773170+0000","last_peered":"2021-01-15T18:49:45.773170+0000","last_clean":"2021-01-15T18:49:45.773170+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:45.773170+0000","last_undegraded":"2021-01-15T18:49:45.773170+0000","last_fullsized":"2021-01-15T18:49:45.773170+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8632,"ondisk_log_size":8632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7659686,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"52'8579","reported_seq":"11486","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:44.835908+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:44.835908+0000","last_peered":"2021-01-15T18:49:44.835908+0000","last_clean":"2021-01-15T18:49:44.835908+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:44.835908+0000","last_undegraded":"2021-01-15T18:49:44.835908+0000","last_fullsized":"2021-01-15T18:49:44.835908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8579,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7340198,"num_objects":897,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8992","reported_seq":"9229","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.870274+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:45.870274+0000","last_peered":"2021-01-15T18:49:45.870274+0000","last_clean":"2021-01-15T18:49:45.870274+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:45.870274+0000","last_undegraded":"2021-01-15T18:49:45.870274+0000","last_fullsized":"2021-01-15T18:49:45.870274+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8992,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7323814,"num_objects":895,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'9099","reported_seq":"9377","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.716446+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:45.716446+0000","last_peered":"2021-01-15T18:49:45.716446+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:45.716446+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9099,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7725388,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"52'8656","reported_seq":"11532","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:44.029010+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:44.029010+0000","last_peered":"2021-01-15T18:49:44.029010+0000","last_clean":"2021-01-15T18:49:44.029010+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:44.029010+0000","last_undegraded":"2021-01-15T18:49:44.029010+0000","last_fullsized":"2021-01-15T18:49:44.029010+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8656,"ondisk_log_size":8656,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7233702,"num_objects":884,"num_object_clones":0,"num_object_copies":3536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":121882924,"num_objects":14893,"num_object_clones":0,"num_object_copies":59572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139832,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17411,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245252096,"data_stored":245252096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139859,"ondisk_log_size":139859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334816,"kb_used_data":59820,"kb_used_omap":0,"kb_used_meta":274943,"kb_avail":94037024,"statfs":{"total":96636764160,"available":96293912576,"internally_reserved":0,"allocated":61255680,"data_stored":60982558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":281541684},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,6,6,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":7.9779999999999998},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":7.944}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":8.1549999999999994},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":8.2460000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":8.3979999999999997},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":8.4580000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335716,"kb_used_data":61244,"kb_used_omap":0,"kb_used_meta":274432,"kb_avail":94036124,"statfs":{"total":96636764160,"available":96292990976,"internally_reserved":0,"allocated":62713856,"data_stored":62440734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":281018368},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,10,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.75},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":1.4510000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":6.3789999999999996},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":6.0110000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331284,"kb_used_data":60700,"kb_used_omap":0,"kb_used_meta":270527,"kb_avail":94040556,"statfs":{"total":96636764160,"available":96297529344,"internally_reserved":0,"allocated":62156800,"data_stored":61883678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":277019700},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":5.085},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":5.1760000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":4.3600000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.012},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.72799999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324456,"kb_used_data":59860,"kb_used_omap":0,"kb_used_meta":264575,"kb_avail":94047384,"statfs":{"total":96636764160,"available":96304521216,"internally_reserved":0,"allocated":61296640,"data_stored":61023518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270924852},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.1040000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.6160000000000001},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.999}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.206},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.2969999999999999}]}]}]}} 2021-01-15T18:49:48.450 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:49.466 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:49.466 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3996/59572 objects degraded (6.708%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":639,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14893,"data_bytes":121882924,"bytes_used":1358102528,"bytes_avail":385188954112,"bytes_total":386547056640,"degraded_objects":3996,"degraded_total":59572,"degraded_ratio":0.067078493251863297,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":164},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:49.493 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:49.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:50.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:50.169 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:50.205 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2021-01-15T18:49:49.936735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120416556,"num_objects":14714,"num_object_clones":0,"num_object_copies":58856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140011,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17411,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140038,"ondisk_log_size":140038,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1301808,"kb_used_data":234292,"kb_used_omap":2,"kb_used_meta":1067325,"kb_avail":376185552,"statfs":{"total":386547056640,"available":385214005248,"internally_reserved":0,"allocated":239915008,"data_stored":238817489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1092940956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,10,11,14,18,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":132,"apply_latency_ms":132,"commit_latency_ns":132000000,"apply_latency_ns":132000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16031744,"num_objects":-1957,"num_object_clones":0,"num_object_copies":-7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1957,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.283505"},"pg_stats":[{"pgid":"1.f","version":"52'8540","reported_seq":"8810","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.665983+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:45.665983+0000","last_peered":"2021-01-15T18:49:45.665983+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:45.665983+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8540,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7929856,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"52'8532","reported_seq":"11468","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.014773+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:46.014773+0000","last_peered":"2021-01-15T18:49:46.014773+0000","last_clean":"2021-01-15T18:49:46.014773+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:46.014773+0000","last_undegraded":"2021-01-15T18:49:46.014773+0000","last_fullsized":"2021-01-15T18:49:46.014773+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8532,"ondisk_log_size":8532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7610534,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"53'8640","reported_seq":"11584","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:48.982985+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:48.982985+0000","last_peered":"2021-01-15T18:49:48.982985+0000","last_clean":"2021-01-15T18:49:48.982985+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:48.982985+0000","last_undegraded":"2021-01-15T18:49:48.982985+0000","last_fullsized":"2021-01-15T18:49:48.982985+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8640,"ondisk_log_size":8640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6975595,"num_objects":853,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8639,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"52'8858","reported_seq":"9116","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.859748+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:45.859748+0000","last_peered":"2021-01-15T18:49:45.859748+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:45.859748+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8858,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7651328,"num_objects":934,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"52'8892","reported_seq":"9496","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.740908+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:45.740908+0000","last_peered":"2021-01-15T18:49:45.740908+0000","last_clean":"2021-01-15T18:49:45.740908+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:45.740908+0000","last_undegraded":"2021-01-15T18:49:45.740908+0000","last_fullsized":"2021-01-15T18:49:45.740908+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8892,"ondisk_log_size":8892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7667878,"num_objects":937,"num_object_clones":0,"num_object_copies":3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"52'8597","reported_seq":"8820","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.080611+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:46.080611+0000","last_peered":"2021-01-15T18:49:46.080611+0000","last_clean":"2021-01-15T18:49:46.080611+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:46.080611+0000","last_undegraded":"2021-01-15T18:49:46.080611+0000","last_fullsized":"2021-01-15T18:49:46.080611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8597,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7610534,"num_objects":930,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"52'8748","reported_seq":"11620","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.974355+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:45.974355+0000","last_peered":"2021-01-15T18:49:45.974355+0000","last_clean":"2021-01-15T18:49:45.974355+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:45.974355+0000","last_undegraded":"2021-01-15T18:49:45.974355+0000","last_fullsized":"2021-01-15T18:49:45.974355+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8748,"ondisk_log_size":8748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8003750,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"52'8900","reported_seq":"11909","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.771423+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:45.771423+0000","last_peered":"2021-01-15T18:49:45.771423+0000","last_clean":"2021-01-15T18:49:45.771423+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:45.771423+0000","last_undegraded":"2021-01-15T18:49:45.771423+0000","last_fullsized":"2021-01-15T18:49:45.771423+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8900,"ondisk_log_size":8900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7835731,"num_objects":957,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"52'8830","reported_seq":"11777","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:46.015403+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:46.015403+0000","last_peered":"2021-01-15T18:49:46.015403+0000","last_clean":"2021-01-15T18:49:46.015403+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:46.015403+0000","last_undegraded":"2021-01-15T18:49:46.015403+0000","last_fullsized":"2021-01-15T18:49:46.015403+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7360595,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"52'8830","reported_seq":"11783","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.862165+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:45.862165+0000","last_peered":"2021-01-15T18:49:45.862165+0000","last_clean":"2021-01-15T18:49:45.862165+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:45.862165+0000","last_undegraded":"2021-01-15T18:49:45.862165+0000","last_fullsized":"2021-01-15T18:49:45.862165+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7598329,"num_objects":929,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"52'8602","reported_seq":"11393","reported_epoch":"52","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.857968+0000","last_change":"2021-01-15T18:48:47.234295+0000","last_active":"2021-01-15T18:49:45.857968+0000","last_peered":"2021-01-15T18:49:45.857968+0000","last_clean":"2021-01-15T18:39:33.411185+0000","last_became_active":"2021-01-15T18:40:32.762013+0000","last_became_peered":"2021-01-15T18:40:32.762013+0000","last_unstale":"2021-01-15T18:49:45.857968+0000","last_undegraded":"2021-01-15T18:40:32.517755+0000","last_fullsized":"2021-01-15T18:40:32.481524+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8602,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7798950,"num_objects":953,"num_object_clones":0,"num_object_copies":3812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'8632","reported_seq":"11473","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.773170+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:45.773170+0000","last_peered":"2021-01-15T18:49:45.773170+0000","last_clean":"2021-01-15T18:49:45.773170+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:45.773170+0000","last_undegraded":"2021-01-15T18:49:45.773170+0000","last_fullsized":"2021-01-15T18:49:45.773170+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8632,"ondisk_log_size":8632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7659686,"num_objects":936,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"53'8633","reported_seq":"11541","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:49.759377+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:49.759377+0000","last_peered":"2021-01-15T18:49:49.759377+0000","last_clean":"2021-01-15T18:49:49.759377+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:49.759377+0000","last_undegraded":"2021-01-15T18:49:49.759377+0000","last_fullsized":"2021-01-15T18:49:49.759377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8633,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6897830,"num_objects":843,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"52'8992","reported_seq":"9229","reported_epoch":"52","state":"active+clean","last_fresh":"2021-01-15T18:49:45.870274+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:45.870274+0000","last_peered":"2021-01-15T18:49:45.870274+0000","last_clean":"2021-01-15T18:49:45.870274+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:45.870274+0000","last_undegraded":"2021-01-15T18:49:45.870274+0000","last_fullsized":"2021-01-15T18:49:45.870274+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8992,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7323814,"num_objects":895,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"52'9099","reported_seq":"9377","reported_epoch":"52","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:45.716446+0000","last_change":"2021-01-15T18:40:32.774717+0000","last_active":"2021-01-15T18:49:45.716446+0000","last_peered":"2021-01-15T18:49:45.716446+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:45.716446+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9099,"ondisk_log_size":9099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7725388,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"53'8713","reported_seq":"11591","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:49.167353+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:49.167353+0000","last_peered":"2021-01-15T18:49:49.167353+0000","last_clean":"2021-01-15T18:49:49.167353+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:49.167353+0000","last_undegraded":"2021-01-15T18:49:49.167353+0000","last_fullsized":"2021-01-15T18:49:49.167353+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8713,"ondisk_log_size":8713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6766758,"num_objects":827,"num_object_clones":0,"num_object_copies":3308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120416556,"num_objects":14714,"num_object_clones":0,"num_object_copies":58856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140011,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17411,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237367296,"data_stored":237367296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140038,"ondisk_log_size":140038,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215227,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":334816,"kb_used_data":59820,"kb_used_omap":0,"kb_used_meta":274943,"kb_avail":94037024,"statfs":{"total":96636764160,"available":96293912576,"internally_reserved":0,"allocated":61255680,"data_stored":60982558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":281541684},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,6,6,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":7.9779999999999998},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":7.944}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":8.1549999999999994},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":8.2460000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":8.3979999999999997},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":8.4580000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321252,"kb_used_data":57212,"kb_used_omap":0,"kb_used_meta":264000,"kb_avail":94050588,"statfs":{"total":96636764160,"available":96307802112,"internally_reserved":0,"allocated":58585088,"data_stored":58306935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":270336000},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,1,0,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.68400000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":1.131}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.843},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.84399999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321284,"kb_used_data":57400,"kb_used_omap":0,"kb_used_meta":263807,"kb_avail":94050556,"statfs":{"total":96636764160,"available":96307769344,"internally_reserved":0,"allocated":58777600,"data_stored":58504478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270138420},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":2.3010000000000002},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":2.3999999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.68400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.873},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.58399999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324456,"kb_used_data":59860,"kb_used_omap":0,"kb_used_meta":264575,"kb_avail":94047384,"statfs":{"total":96636764160,"available":96304521216,"internally_reserved":0,"allocated":61296640,"data_stored":61023518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270924852},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.0840000000000001,"5min":3.7770000000000001,"15min":3.5070000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":19.954000000000001,"5min":30.905999999999999,"15min":35.850999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.9530000000000001,"15min":2.0550000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":6.9279999999999999,"5min":21.649999999999999,"15min":32.216999999999999},"last":1.1040000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":3.5129999999999999,"5min":2.6080000000000001,"15min":2.7229999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":24.457999999999998,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.6160000000000001},{"interface":"front","average":{"1min":2.2949999999999999,"5min":2.242,"15min":1.9490000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":11.984,"5min":12.609999999999999,"15min":19.381},"last":0.999}]},{"osd":3,"last update":"Fri Jan 15 18:48:48 2021","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.923,"15min":3.6739999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":13.170999999999999,"5min":25.216000000000001,"15min":25.216000000000001},"last":1.206},{"interface":"front","average":{"1min":2.8889999999999998,"5min":2.831,"15min":2.661},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":13.256,"5min":17.478000000000002,"15min":17.478000000000002},"last":1.2969999999999999}]}]}]}} 2021-01-15T18:49:53.207 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:54.241 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:54.250 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:54.277 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2021-01-15T18:49:52.657854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114551084,"num_objects":13998,"num_object_clones":0,"num_object_copies":55992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140727,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17490,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140754,"ondisk_log_size":140754,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1281836,"kb_used_data":227348,"kb_used_omap":2,"kb_used_meta":1054269,"kb_avail":376205524,"statfs":{"total":386547056640,"available":385234456576,"internally_reserved":0,"allocated":232804352,"data_stored":231686667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1079571612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,2,20,16,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14344192,"num_objects":-1751,"num_object_clones":0,"num_object_copies":-7004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1751,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003722"},"pg_stats":[{"pgid":"1.f","version":"54'8582","reported_seq":"8854","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:50.382029+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:50.382029+0000","last_peered":"2021-01-15T18:49:50.382029+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:50.382029+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8582,"ondisk_log_size":8582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7585792,"num_objects":926,"num_object_clones":0,"num_object_copies":3704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8598","reported_seq":"11536","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:51.126135+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:51.126135+0000","last_peered":"2021-01-15T18:49:51.126135+0000","last_clean":"2021-01-15T18:49:51.126135+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:51.126135+0000","last_undegraded":"2021-01-15T18:49:51.126135+0000","last_fullsized":"2021-01-15T18:49:51.126135+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8598,"ondisk_log_size":8598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7069862,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"53'8640","reported_seq":"11584","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:48.982985+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:48.982985+0000","last_peered":"2021-01-15T18:49:48.982985+0000","last_clean":"2021-01-15T18:49:48.982985+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:48.982985+0000","last_undegraded":"2021-01-15T18:49:48.982985+0000","last_fullsized":"2021-01-15T18:49:48.982985+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8640,"ondisk_log_size":8640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6975595,"num_objects":853,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8639,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'8922","reported_seq":"9181","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:50.867689+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:50.867689+0000","last_peered":"2021-01-15T18:49:50.867689+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:50.867689+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8922,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7127040,"num_objects":870,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'8952","reported_seq":"9556","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:51.103050+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:51.103050+0000","last_peered":"2021-01-15T18:49:51.103050+0000","last_clean":"2021-01-15T18:49:51.103050+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:51.103050+0000","last_undegraded":"2021-01-15T18:49:51.103050+0000","last_fullsized":"2021-01-15T18:49:51.103050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8952,"ondisk_log_size":8952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7176358,"num_objects":877,"num_object_clones":0,"num_object_copies":3508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8646","reported_seq":"8871","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.772606+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:50.772606+0000","last_peered":"2021-01-15T18:49:50.772606+0000","last_clean":"2021-01-15T18:49:50.772606+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:50.772606+0000","last_undegraded":"2021-01-15T18:49:50.772606+0000","last_fullsized":"2021-01-15T18:49:50.772606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8646,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7209126,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8644,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8798","reported_seq":"11673","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:51.016743+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:51.016743+0000","last_peered":"2021-01-15T18:49:51.016743+0000","last_clean":"2021-01-15T18:49:51.016743+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:51.016743+0000","last_undegraded":"2021-01-15T18:49:51.016743+0000","last_fullsized":"2021-01-15T18:49:51.016743+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8798,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7594150,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'8958","reported_seq":"11969","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.863874+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:50.863874+0000","last_peered":"2021-01-15T18:49:50.863874+0000","last_clean":"2021-01-15T18:49:50.863874+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:50.863874+0000","last_undegraded":"2021-01-15T18:49:50.863874+0000","last_fullsized":"2021-01-15T18:49:50.863874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8958,"ondisk_log_size":8958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7360595,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8957,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'8866","reported_seq":"11815","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.893932+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:50.893932+0000","last_peered":"2021-01-15T18:49:50.893932+0000","last_clean":"2021-01-15T18:49:50.893932+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:50.893932+0000","last_undegraded":"2021-01-15T18:49:50.893932+0000","last_fullsized":"2021-01-15T18:49:50.893932+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8866,"ondisk_log_size":8866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7065683,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'8883","reported_seq":"11837","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.522177+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:50.522177+0000","last_peered":"2021-01-15T18:49:50.522177+0000","last_clean":"2021-01-15T18:49:50.522177+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:50.522177+0000","last_undegraded":"2021-01-15T18:49:50.522177+0000","last_fullsized":"2021-01-15T18:49:50.522177+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8883,"ondisk_log_size":8883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7164153,"num_objects":876,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8669","reported_seq":"11522","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.606120+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:49:50.606120+0000","last_peered":"2021-01-15T18:49:50.606120+0000","last_clean":"2021-01-15T18:49:50.606120+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:49:50.606120+0000","last_undegraded":"2021-01-15T18:49:50.606120+0000","last_fullsized":"2021-01-15T18:49:50.606120+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8669,"ondisk_log_size":8669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7250086,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8667,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8694","reported_seq":"11537","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:51.115922+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:51.115922+0000","last_peered":"2021-01-15T18:49:51.115922+0000","last_clean":"2021-01-15T18:49:51.115922+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:51.115922+0000","last_undegraded":"2021-01-15T18:49:51.115922+0000","last_fullsized":"2021-01-15T18:49:51.115922+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8694,"ondisk_log_size":8694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7151782,"num_objects":874,"num_object_clones":0,"num_object_copies":3496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"53'8633","reported_seq":"11541","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:49.759377+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:49.759377+0000","last_peered":"2021-01-15T18:49:49.759377+0000","last_clean":"2021-01-15T18:49:49.759377+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:49.759377+0000","last_undegraded":"2021-01-15T18:49:49.759377+0000","last_fullsized":"2021-01-15T18:49:49.759377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8633,"ondisk_log_size":8633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6897830,"num_objects":843,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9049","reported_seq":"9289","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:50.941122+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:50.941122+0000","last_peered":"2021-01-15T18:49:50.941122+0000","last_clean":"2021-01-15T18:49:50.941122+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:50.941122+0000","last_undegraded":"2021-01-15T18:49:50.941122+0000","last_fullsized":"2021-01-15T18:49:50.941122+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9049,"ondisk_log_size":9049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6856870,"num_objects":838,"num_object_clones":0,"num_object_copies":3352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9047,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9151","reported_seq":"9541","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:50.916524+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:49:50.916524+0000","last_peered":"2021-01-15T18:49:50.916524+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:50.916524+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9151,"ondisk_log_size":9151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7299404,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"53'8713","reported_seq":"11591","reported_epoch":"53","state":"active+clean","last_fresh":"2021-01-15T18:49:49.167353+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:49.167353+0000","last_peered":"2021-01-15T18:49:49.167353+0000","last_clean":"2021-01-15T18:49:49.167353+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:49.167353+0000","last_undegraded":"2021-01-15T18:49:49.167353+0000","last_fullsized":"2021-01-15T18:49:49.167353+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8713,"ondisk_log_size":8713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6766758,"num_objects":827,"num_object_clones":0,"num_object_copies":3308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":114551084,"num_objects":13998,"num_object_clones":0,"num_object_copies":55992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140727,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17490,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230137856,"data_stored":230137856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140754,"ondisk_log_size":140754,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215228,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324768,"kb_used_data":56364,"kb_used_omap":0,"kb_used_meta":268351,"kb_avail":94047072,"statfs":{"total":96636764160,"available":96304201728,"internally_reserved":0,"allocated":57716736,"data_stored":57433531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":274791476},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":1.024},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.58099999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":5.2839999999999998},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":10.154}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":25.425000000000001},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":5.7300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321252,"kb_used_data":57212,"kb_used_omap":0,"kb_used_meta":264000,"kb_avail":94050588,"statfs":{"total":96636764160,"available":96307802112,"internally_reserved":0,"allocated":58585088,"data_stored":58306935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":270336000},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,1,0,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.68400000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":1.131}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":1.843},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.84399999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":321284,"kb_used_data":57400,"kb_used_omap":0,"kb_used_meta":263807,"kb_avail":94050556,"statfs":{"total":96636764160,"available":96307769344,"internally_reserved":0,"allocated":58777600,"data_stored":58504478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":270138420},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":2.3010000000000002},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":2.3999999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.68400000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.873},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.58399999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314532,"kb_used_data":56372,"kb_used_omap":0,"kb_used_meta":258111,"kb_avail":94057308,"statfs":{"total":96636764160,"available":96314683392,"internally_reserved":0,"allocated":57724928,"data_stored":57441723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":264305716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":3.4550000000000001},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":13.401}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":3.4079999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":3.5659999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":3.492}]}]}]}} 2021-01-15T18:49:54.279 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:49:55.603 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:55.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3917/55544 objects degraded (7.052%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":646,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13886,"data_bytes":113633580,"bytes_used":1319161856,"bytes_avail":385227894784,"bytes_total":386547056640,"degraded_objects":3917,"degraded_total":55544,"degraded_ratio":0.070520668299006189,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":125},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:49:55.766 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:49:55.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:49:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:49:57.149 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:49:57.174 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2021-01-15T18:49:56.658681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107104556,"num_objects":13089,"num_object_clones":0,"num_object_copies":52356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141636,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17527,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141663,"ondisk_log_size":141663,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1302984,"kb_used_data":213292,"kb_used_omap":2,"kb_used_meta":1089533,"kb_avail":376184376,"statfs":{"total":386547056640,"available":385212801024,"internally_reserved":0,"allocated":218411008,"data_stored":217278188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1115681948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,6,7,23,33,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":134,"apply_latency_ms":134,"commit_latency_ns":134000000,"apply_latency_ns":134000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14778368,"num_objects":-1804,"num_object_clones":0,"num_object_copies":-7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003684"},"pg_stats":[{"pgid":"1.f","version":"54'8633","reported_seq":"8905","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.462368+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:55.462368+0000","last_peered":"2021-01-15T18:49:55.462368+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:55.462368+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8633,"ondisk_log_size":8633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7168000,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8633,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8652","reported_seq":"11589","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.005374+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:56.005374+0000","last_peered":"2021-01-15T18:49:56.005374+0000","last_clean":"2021-01-15T18:49:56.005374+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:56.005374+0000","last_undegraded":"2021-01-15T18:49:56.005374+0000","last_fullsized":"2021-01-15T18:49:56.005374+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8652,"ondisk_log_size":8652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6627494,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8695","reported_seq":"11640","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:54.327339+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:54.327339+0000","last_peered":"2021-01-15T18:49:54.327339+0000","last_clean":"2021-01-15T18:49:54.327339+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:54.327339+0000","last_undegraded":"2021-01-15T18:49:54.327339+0000","last_fullsized":"2021-01-15T18:49:54.327339+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8695,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6525035,"num_objects":798,"num_object_clones":0,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8694,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'8980","reported_seq":"9240","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.908439+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:55.908439+0000","last_peered":"2021-01-15T18:49:55.908439+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:55.908439+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8980,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6651904,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'8998","reported_seq":"9604","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.005049+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:56.005049+0000","last_peered":"2021-01-15T18:49:56.005049+0000","last_clean":"2021-01-15T18:49:56.005049+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:56.005049+0000","last_undegraded":"2021-01-15T18:49:56.005049+0000","last_fullsized":"2021-01-15T18:49:56.005049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8998,"ondisk_log_size":8998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6799526,"num_objects":831,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8996,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8705","reported_seq":"8930","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.062045+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:56.062045+0000","last_peered":"2021-01-15T18:49:56.062045+0000","last_clean":"2021-01-15T18:49:56.062045+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:56.062045+0000","last_undegraded":"2021-01-15T18:49:56.062045+0000","last_fullsized":"2021-01-15T18:49:56.062045+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8705,"ondisk_log_size":8705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6725798,"num_objects":822,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8859","reported_seq":"11733","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.072624+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:56.072624+0000","last_peered":"2021-01-15T18:49:56.072624+0000","last_clean":"2021-01-15T18:49:56.072624+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:56.072624+0000","last_undegraded":"2021-01-15T18:49:56.072624+0000","last_fullsized":"2021-01-15T18:49:56.072624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8859,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7094438,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9018","reported_seq":"12029","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.857034+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:55.857034+0000","last_peered":"2021-01-15T18:49:55.857034+0000","last_clean":"2021-01-15T18:49:55.857034+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:55.857034+0000","last_undegraded":"2021-01-15T18:49:55.857034+0000","last_fullsized":"2021-01-15T18:49:55.857034+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9018,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6869075,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'8933","reported_seq":"11882","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.065881+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:56.065881+0000","last_peered":"2021-01-15T18:49:56.065881+0000","last_clean":"2021-01-15T18:49:56.065881+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:56.065881+0000","last_undegraded":"2021-01-15T18:49:56.065881+0000","last_fullsized":"2021-01-15T18:49:56.065881+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8933,"ondisk_log_size":8933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6516819,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'8939","reported_seq":"11894","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.822673+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:55.822673+0000","last_peered":"2021-01-15T18:49:55.822673+0000","last_clean":"2021-01-15T18:49:55.822673+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:55.822673+0000","last_undegraded":"2021-01-15T18:49:55.822673+0000","last_fullsized":"2021-01-15T18:49:55.822673+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8939,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6705401,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8723","reported_seq":"11575","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.905925+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:49:55.905925+0000","last_peered":"2021-01-15T18:49:55.905925+0000","last_clean":"2021-01-15T18:49:55.905925+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:49:55.905925+0000","last_undegraded":"2021-01-15T18:49:55.905925+0000","last_fullsized":"2021-01-15T18:49:55.905925+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8723,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6807718,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8748","reported_seq":"11590","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.954077+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:55.954077+0000","last_peered":"2021-01-15T18:49:55.954077+0000","last_clean":"2021-01-15T18:49:55.954077+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:55.954077+0000","last_undegraded":"2021-01-15T18:49:55.954077+0000","last_fullsized":"2021-01-15T18:49:55.954077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8748,"ondisk_log_size":8748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6709414,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8699","reported_seq":"11608","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:54.725379+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:54.725379+0000","last_peered":"2021-01-15T18:49:54.725379+0000","last_clean":"2021-01-15T18:49:54.725379+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:54.725379+0000","last_undegraded":"2021-01-15T18:49:54.725379+0000","last_fullsized":"2021-01-15T18:49:54.725379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8699,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6357158,"num_objects":777,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9107","reported_seq":"9347","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.856753+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:55.856753+0000","last_peered":"2021-01-15T18:49:55.856753+0000","last_clean":"2021-01-15T18:49:55.856753+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:55.856753+0000","last_undegraded":"2021-01-15T18:49:55.856753+0000","last_fullsized":"2021-01-15T18:49:55.856753+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9107,"ondisk_log_size":9107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6381734,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9204","reported_seq":"9669","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.847185+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:49:55.847185+0000","last_peered":"2021-01-15T18:49:55.847185+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:55.847185+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9204,"ondisk_log_size":9204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6865228,"num_objects":840,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9201,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8770","reported_seq":"11649","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:54.245805+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:54.245805+0000","last_peered":"2021-01-15T18:49:54.245805+0000","last_clean":"2021-01-15T18:49:54.245805+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:54.245805+0000","last_undegraded":"2021-01-15T18:49:54.245805+0000","last_fullsized":"2021-01-15T18:49:54.245805+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8770,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6299814,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":107104556,"num_objects":13089,"num_object_clones":0,"num_object_copies":52356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141636,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17527,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215429120,"data_stored":215429120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141663,"ondisk_log_size":141663,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326876,"kb_used_data":52680,"kb_used_omap":0,"kb_used_meta":274111,"kb_avail":94044964,"statfs":{"total":96636764160,"available":96302043136,"internally_reserved":0,"allocated":53944320,"data_stored":53661115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":280689716},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,8,8,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":1.147},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.357},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":1.5289999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.04},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":2.0419999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327660,"kb_used_data":53900,"kb_used_omap":0,"kb_used_meta":273792,"kb_avail":94044180,"statfs":{"total":96636764160,"available":96301240320,"internally_reserved":0,"allocated":55193600,"data_stored":54910395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":280363008},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.748}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":4.2960000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":5.1680000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327728,"kb_used_data":54088,"kb_used_omap":0,"kb_used_meta":273599,"kb_avail":94044112,"statfs":{"total":96636764160,"available":96301170688,"internally_reserved":0,"allocated":55386112,"data_stored":55102907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":280165428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":7.7569999999999997},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":7.8609999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":2.7130000000000001},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":2.4140000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.8470000000000004},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.48699999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320720,"kb_used_data":52624,"kb_used_omap":0,"kb_used_meta":268031,"kb_avail":94051120,"statfs":{"total":96636764160,"available":96308346880,"internally_reserved":0,"allocated":53886976,"data_stored":53603771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":274463796},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.5289999999999999},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.6819999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.69599999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89300000000000002}]}]}]}} 2021-01-15T18:50:00.177 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:01.350 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:01.385 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:01.453 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2021-01-15T18:50:00.659489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105589036,"num_objects":12904,"num_object_clones":0,"num_object_copies":51616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141821,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17527,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141848,"ondisk_log_size":141848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1311024,"kb_used_data":205620,"kb_used_omap":2,"kb_used_meta":1105213,"kb_avail":376176336,"statfs":{"total":386547056640,"available":385204568064,"internally_reserved":0,"allocated":210554880,"data_stored":209422060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1131738268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,10,15,18,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":139,"apply_latency_ms":139,"commit_latency_ns":139000000,"apply_latency_ns":139000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14827520,"num_objects":-1810,"num_object_clones":0,"num_object_copies":-7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1810,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.722753"},"pg_stats":[{"pgid":"1.f","version":"54'8633","reported_seq":"8905","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.462368+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:49:55.462368+0000","last_peered":"2021-01-15T18:49:55.462368+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:49:55.462368+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8633,"ondisk_log_size":8633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7168000,"num_objects":875,"num_object_clones":0,"num_object_copies":3500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8633,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-375}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8652","reported_seq":"11589","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.005374+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:49:56.005374+0000","last_peered":"2021-01-15T18:49:56.005374+0000","last_clean":"2021-01-15T18:49:56.005374+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:49:56.005374+0000","last_undegraded":"2021-01-15T18:49:56.005374+0000","last_fullsized":"2021-01-15T18:49:56.005374+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8652,"ondisk_log_size":8652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6627494,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8761","reported_seq":"11706","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.263420+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:59.263420+0000","last_peered":"2021-01-15T18:49:59.263420+0000","last_clean":"2021-01-15T18:49:59.263420+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:59.263420+0000","last_undegraded":"2021-01-15T18:49:59.263420+0000","last_fullsized":"2021-01-15T18:49:59.263420+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8761,"ondisk_log_size":8761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5984363,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8760,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'8980","reported_seq":"9240","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.908439+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:49:55.908439+0000","last_peered":"2021-01-15T18:49:55.908439+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:49:55.908439+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8980,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6651904,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'8998","reported_seq":"9604","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.005049+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:49:56.005049+0000","last_peered":"2021-01-15T18:49:56.005049+0000","last_clean":"2021-01-15T18:49:56.005049+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:49:56.005049+0000","last_undegraded":"2021-01-15T18:49:56.005049+0000","last_fullsized":"2021-01-15T18:49:56.005049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8998,"ondisk_log_size":8998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6799526,"num_objects":831,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8996,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8705","reported_seq":"8930","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.062045+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:49:56.062045+0000","last_peered":"2021-01-15T18:49:56.062045+0000","last_clean":"2021-01-15T18:49:56.062045+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:49:56.062045+0000","last_undegraded":"2021-01-15T18:49:56.062045+0000","last_fullsized":"2021-01-15T18:49:56.062045+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8705,"ondisk_log_size":8705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6725798,"num_objects":822,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8859","reported_seq":"11733","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.072624+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:49:56.072624+0000","last_peered":"2021-01-15T18:49:56.072624+0000","last_clean":"2021-01-15T18:49:56.072624+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:49:56.072624+0000","last_undegraded":"2021-01-15T18:49:56.072624+0000","last_fullsized":"2021-01-15T18:49:56.072624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8859,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7094438,"num_objects":867,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9018","reported_seq":"12029","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.857034+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:49:55.857034+0000","last_peered":"2021-01-15T18:49:55.857034+0000","last_clean":"2021-01-15T18:49:55.857034+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:49:55.857034+0000","last_undegraded":"2021-01-15T18:49:55.857034+0000","last_fullsized":"2021-01-15T18:49:55.857034+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9018,"ondisk_log_size":9018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6869075,"num_objects":839,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'8933","reported_seq":"11882","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:56.065881+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:49:56.065881+0000","last_peered":"2021-01-15T18:49:56.065881+0000","last_clean":"2021-01-15T18:49:56.065881+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:49:56.065881+0000","last_undegraded":"2021-01-15T18:49:56.065881+0000","last_fullsized":"2021-01-15T18:49:56.065881+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8933,"ondisk_log_size":8933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6516819,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'8939","reported_seq":"11894","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.822673+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:49:55.822673+0000","last_peered":"2021-01-15T18:49:55.822673+0000","last_clean":"2021-01-15T18:49:55.822673+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:49:55.822673+0000","last_undegraded":"2021-01-15T18:49:55.822673+0000","last_fullsized":"2021-01-15T18:49:55.822673+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8939,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6705401,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8723","reported_seq":"11575","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.905925+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:49:55.905925+0000","last_peered":"2021-01-15T18:49:55.905925+0000","last_clean":"2021-01-15T18:49:55.905925+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:49:55.905925+0000","last_undegraded":"2021-01-15T18:49:55.905925+0000","last_fullsized":"2021-01-15T18:49:55.905925+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8723,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6807718,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8748","reported_seq":"11590","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.954077+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:49:55.954077+0000","last_peered":"2021-01-15T18:49:55.954077+0000","last_clean":"2021-01-15T18:49:55.954077+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:49:55.954077+0000","last_undegraded":"2021-01-15T18:49:55.954077+0000","last_fullsized":"2021-01-15T18:49:55.954077+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8748,"ondisk_log_size":8748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6709414,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8758","reported_seq":"11667","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.875743+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:59.875743+0000","last_peered":"2021-01-15T18:49:59.875743+0000","last_clean":"2021-01-15T18:49:59.875743+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:59.875743+0000","last_undegraded":"2021-01-15T18:49:59.875743+0000","last_fullsized":"2021-01-15T18:49:59.875743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8758,"ondisk_log_size":8758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5873830,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9107","reported_seq":"9347","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:55.856753+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:49:55.856753+0000","last_peered":"2021-01-15T18:49:55.856753+0000","last_clean":"2021-01-15T18:49:55.856753+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:49:55.856753+0000","last_undegraded":"2021-01-15T18:49:55.856753+0000","last_fullsized":"2021-01-15T18:49:55.856753+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9107,"ondisk_log_size":9107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6381734,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9204","reported_seq":"9669","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:49:55.847185+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:49:55.847185+0000","last_peered":"2021-01-15T18:49:55.847185+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:49:55.847185+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9204,"ondisk_log_size":9204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6865228,"num_objects":840,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9201,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8830","reported_seq":"11709","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.349934+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:59.349934+0000","last_peered":"2021-01-15T18:49:59.349934+0000","last_clean":"2021-01-15T18:49:59.349934+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:59.349934+0000","last_undegraded":"2021-01-15T18:49:59.349934+0000","last_fullsized":"2021-01-15T18:49:59.349934+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5808294,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":105589036,"num_objects":12904,"num_object_clones":0,"num_object_copies":51616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141821,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17527,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207585280,"data_stored":207585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141848,"ondisk_log_size":141848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326876,"kb_used_data":52680,"kb_used_omap":0,"kb_used_meta":274111,"kb_avail":94044964,"statfs":{"total":96636764160,"available":96302043136,"internally_reserved":0,"allocated":53944320,"data_stored":53661115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":280689716},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,8,8,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":1.147},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.357},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":1.5289999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.04},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":2.0419999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333748,"kb_used_data":50084,"kb_used_omap":0,"kb_used_meta":283648,"kb_avail":94038092,"statfs":{"total":96636764160,"available":96295006208,"internally_reserved":0,"allocated":51286016,"data_stored":51002811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":290455552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.298}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":5.5529999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.625},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.72899999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329680,"kb_used_data":50232,"kb_used_omap":0,"kb_used_meta":279423,"kb_avail":94042160,"statfs":{"total":96636764160,"available":96299171840,"internally_reserved":0,"allocated":51437568,"data_stored":51154363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":286129204},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":1.4330000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.746}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.109},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.72099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.575},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.81299999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320720,"kb_used_data":52624,"kb_used_omap":0,"kb_used_meta":268031,"kb_avail":94051120,"statfs":{"total":96636764160,"available":96308346880,"internally_reserved":0,"allocated":53886976,"data_stored":53603771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":274463796},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.5289999999999999},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.6819999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.69599999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89300000000000002}]}]}]}} 2021-01-15T18:50:01.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:02.778 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:02.778 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3880/51616 objects degraded (7.517%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":653,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12904,"data_bytes":105589036,"bytes_used":1342488576,"bytes_avail":385204568064,"bytes_total":386547056640,"degraded_objects":3880,"degraded_total":51616,"degraded_ratio":0.07517048977061376,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:02.882 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:02.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:03.913 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:03.938 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2021-01-15T18:50:02.659961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99002668,"num_objects":12100,"num_object_clones":0,"num_object_copies":48400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142625,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17571,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142652,"ondisk_log_size":142652,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1302704,"kb_used_data":197956,"kb_used_omap":2,"kb_used_meta":1104637,"kb_avail":376184656,"statfs":{"total":386547056640,"available":385213087744,"internally_reserved":0,"allocated":202706944,"data_stored":201574124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1131148444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,13,11,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":131,"apply_latency_ms":131,"commit_latency_ns":131000000,"apply_latency_ns":131000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21413888,"num_objects":-2614,"num_object_clones":0,"num_object_copies":-10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004571"},"pg_stats":[{"pgid":"1.f","version":"54'8715","reported_seq":"8986","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:01.000793+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:01.000793+0000","last_peered":"2021-01-15T18:50:01.000793+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:01.000793+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8715,"ondisk_log_size":8715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6496256,"num_objects":793,"num_object_clones":0,"num_object_copies":3172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8714","reported_seq":"11652","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.185660+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:01.185660+0000","last_peered":"2021-01-15T18:50:01.185660+0000","last_clean":"2021-01-15T18:50:01.185660+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:01.185660+0000","last_undegraded":"2021-01-15T18:50:01.185660+0000","last_fullsized":"2021-01-15T18:50:01.185660+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8714,"ondisk_log_size":8714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6119590,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8712,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8761","reported_seq":"11706","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.263420+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:49:59.263420+0000","last_peered":"2021-01-15T18:49:59.263420+0000","last_clean":"2021-01-15T18:49:59.263420+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:49:59.263420+0000","last_undegraded":"2021-01-15T18:49:59.263420+0000","last_fullsized":"2021-01-15T18:49:59.263420+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8761,"ondisk_log_size":8761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5984363,"num_objects":732,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8760,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9041","reported_seq":"9301","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:00.807963+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:00.807963+0000","last_peered":"2021-01-15T18:50:00.807963+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:00.807963+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9041,"ondisk_log_size":9041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6152192,"num_objects":751,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9053","reported_seq":"9659","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.024557+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:01.024557+0000","last_peered":"2021-01-15T18:50:01.024557+0000","last_clean":"2021-01-15T18:50:01.024557+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:01.024557+0000","last_undegraded":"2021-01-15T18:50:01.024557+0000","last_fullsized":"2021-01-15T18:50:01.024557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9053,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6348966,"num_objects":776,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8768","reported_seq":"8993","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:00.860101+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:00.860101+0000","last_peered":"2021-01-15T18:50:00.860101+0000","last_clean":"2021-01-15T18:50:00.860101+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:00.860101+0000","last_undegraded":"2021-01-15T18:50:00.860101+0000","last_fullsized":"2021-01-15T18:50:00.860101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8768,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6209702,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8914","reported_seq":"11789","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.109074+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:01.109074+0000","last_peered":"2021-01-15T18:50:01.109074+0000","last_clean":"2021-01-15T18:50:01.109074+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:01.109074+0000","last_undegraded":"2021-01-15T18:50:01.109074+0000","last_fullsized":"2021-01-15T18:50:01.109074+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8914,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6643878,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9082","reported_seq":"12090","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.054363+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:01.054363+0000","last_peered":"2021-01-15T18:50:01.054363+0000","last_clean":"2021-01-15T18:50:01.054363+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:01.054363+0000","last_undegraded":"2021-01-15T18:50:01.054363+0000","last_fullsized":"2021-01-15T18:50:01.054363+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9082,"ondisk_log_size":9082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6344787,"num_objects":775,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'8995","reported_seq":"11944","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.129495+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:01.129495+0000","last_peered":"2021-01-15T18:50:01.129495+0000","last_clean":"2021-01-15T18:50:01.129495+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:01.129495+0000","last_undegraded":"2021-01-15T18:50:01.129495+0000","last_fullsized":"2021-01-15T18:50:01.129495+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8995,"ondisk_log_size":8995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6008915,"num_objects":734,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8994,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9001","reported_seq":"11956","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:00.883800+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:00.883800+0000","last_peered":"2021-01-15T18:50:00.883800+0000","last_clean":"2021-01-15T18:50:00.883800+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:00.883800+0000","last_undegraded":"2021-01-15T18:50:00.883800+0000","last_fullsized":"2021-01-15T18:50:00.883800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9001,"ondisk_log_size":9001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6197497,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8785","reported_seq":"11638","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:00.842383+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:00.842383+0000","last_peered":"2021-01-15T18:50:00.842383+0000","last_clean":"2021-01-15T18:50:00.842383+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:00.842383+0000","last_undegraded":"2021-01-15T18:50:00.842383+0000","last_fullsized":"2021-01-15T18:50:00.842383+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8785,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6299814,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8783,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8816","reported_seq":"11658","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:01.148101+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:01.148101+0000","last_peered":"2021-01-15T18:50:01.148101+0000","last_clean":"2021-01-15T18:50:01.148101+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:01.148101+0000","last_undegraded":"2021-01-15T18:50:01.148101+0000","last_fullsized":"2021-01-15T18:50:01.148101+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8816,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6152358,"num_objects":752,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8758","reported_seq":"11667","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.875743+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:49:59.875743+0000","last_peered":"2021-01-15T18:49:59.875743+0000","last_clean":"2021-01-15T18:49:59.875743+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:49:59.875743+0000","last_undegraded":"2021-01-15T18:49:59.875743+0000","last_fullsized":"2021-01-15T18:49:59.875743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8758,"ondisk_log_size":8758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5873830,"num_objects":718,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9154","reported_seq":"9394","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:00.903832+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:00.903832+0000","last_peered":"2021-01-15T18:50:00.903832+0000","last_clean":"2021-01-15T18:50:00.903832+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:00.903832+0000","last_undegraded":"2021-01-15T18:50:00.903832+0000","last_fullsized":"2021-01-15T18:50:00.903832+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9154,"ondisk_log_size":9154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5996710,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9265","reported_seq":"9817","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:01.004431+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:01.004431+0000","last_peered":"2021-01-15T18:50:01.004431+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:01.004431+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9265,"ondisk_log_size":9265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6365516,"num_objects":779,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8830","reported_seq":"11709","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:49:59.349934+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:49:59.349934+0000","last_peered":"2021-01-15T18:49:59.349934+0000","last_clean":"2021-01-15T18:49:59.349934+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:49:59.349934+0000","last_undegraded":"2021-01-15T18:49:59.349934+0000","last_fullsized":"2021-01-15T18:49:59.349934+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5808294,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":99002668,"num_objects":12100,"num_object_clones":0,"num_object_copies":48400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142625,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17571,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199737344,"data_stored":199737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142652,"ondisk_log_size":142652,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215230,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316476,"kb_used_data":48800,"kb_used_omap":0,"kb_used_meta":267647,"kb_avail":94055364,"statfs":{"total":96636764160,"available":96312692736,"internally_reserved":0,"allocated":49971200,"data_stored":49687995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":274070580},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.67700000000000005},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":1.97}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":8.3049999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4079999999999995},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":25.233000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333748,"kb_used_data":50084,"kb_used_omap":0,"kb_used_meta":283648,"kb_avail":94038092,"statfs":{"total":96636764160,"available":96295006208,"internally_reserved":0,"allocated":51286016,"data_stored":51002811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":290455552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":2.7440000000000002,"15min":3.3490000000000002},"min":{"1min":0.39600000000000002,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":6.6630000000000003,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.982,"5min":2.6699999999999999,"15min":1.9259999999999999},"min":{"1min":0.46100000000000002,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":8.0579999999999998,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.298}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":5.5529999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:48:59 2021","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":3.2879999999999998,"15min":2.7469999999999999},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.39200000000000002},"max":{"1min":10.246,"5min":25.335000000000001,"15min":25.335000000000001},"last":0.625},{"interface":"front","average":{"1min":1.6559999999999999,"5min":2.3889999999999998,"15min":2.5670000000000002},"min":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":10.759,"5min":18.34,"15min":18.34},"last":0.72899999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329680,"kb_used_data":50232,"kb_used_omap":0,"kb_used_meta":279423,"kb_avail":94042160,"statfs":{"total":96636764160,"available":96299171840,"internally_reserved":0,"allocated":51437568,"data_stored":51154363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":286129204},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":1.4330000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.746}]},{"osd":2,"last update":"Fri Jan 15 18:49:01 2021","interfaces":[{"interface":"back","average":{"1min":3.673,"5min":3.7669999999999999,"15min":2.9510000000000001},"min":{"1min":0.443,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":17.143000000000001,"5min":25.239999999999998,"15min":27.77},"last":1.109},{"interface":"front","average":{"1min":1.827,"5min":2.399,"15min":2.6230000000000002},"min":{"1min":0.47499999999999998,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":9.2159999999999993,"5min":12.413,"15min":22.765000000000001},"last":0.72099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.575},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":0.81299999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322800,"kb_used_data":48840,"kb_used_omap":0,"kb_used_meta":273919,"kb_avail":94049040,"statfs":{"total":96636764160,"available":96306216960,"internally_reserved":0,"allocated":50012160,"data_stored":49728955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":280493108},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":4.9349999999999996},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.181}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":1.484}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.377}]}]}]}} 2021-01-15T18:50:06.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:08.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:08.260 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:08.294 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2021-01-15T18:50:06.660802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91425068,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143550,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17616,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143577,"ondisk_log_size":143577,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1273932,"kb_used_data":182756,"kb_used_omap":2,"kb_used_meta":1091069,"kb_avail":376213428,"statfs":{"total":386547056640,"available":385242550272,"internally_reserved":0,"allocated":187142144,"data_stored":186009324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1117254812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,15,12,18,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22208512,"num_objects":-2711,"num_object_clones":0,"num_object_copies":-10844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"1.f","version":"54'8782","reported_seq":"9054","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:06.017864+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:06.017864+0000","last_peered":"2021-01-15T18:50:06.017864+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:06.017864+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8782,"ondisk_log_size":8782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5947392,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8774","reported_seq":"11712","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.098919+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:06.098919+0000","last_peered":"2021-01-15T18:50:06.098919+0000","last_clean":"2021-01-15T18:50:06.098919+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:06.098919+0000","last_undegraded":"2021-01-15T18:50:06.098919+0000","last_fullsized":"2021-01-15T18:50:06.098919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8774,"ondisk_log_size":8774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5628070,"num_objects":688,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8804","reported_seq":"11749","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:04.230071+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:04.230071+0000","last_peered":"2021-01-15T18:50:04.230071+0000","last_clean":"2021-01-15T18:50:04.230071+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:04.230071+0000","last_undegraded":"2021-01-15T18:50:04.230071+0000","last_fullsized":"2021-01-15T18:50:04.230071+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8804,"ondisk_log_size":8804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5632107,"num_objects":689,"num_object_clones":0,"num_object_copies":2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8803,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9088","reported_seq":"9348","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:05.874335+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:05.874335+0000","last_peered":"2021-01-15T18:50:05.874335+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:05.874335+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9088,"ondisk_log_size":9088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5767168,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9117","reported_seq":"9723","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.077247+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:06.077247+0000","last_peered":"2021-01-15T18:50:06.077247+0000","last_clean":"2021-01-15T18:50:06.077247+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:06.077247+0000","last_undegraded":"2021-01-15T18:50:06.077247+0000","last_fullsized":"2021-01-15T18:50:06.077247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9117,"ondisk_log_size":9117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5824678,"num_objects":712,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8823","reported_seq":"9048","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.021154+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:06.021154+0000","last_peered":"2021-01-15T18:50:06.021154+0000","last_clean":"2021-01-15T18:50:06.021154+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:06.021154+0000","last_undegraded":"2021-01-15T18:50:06.021154+0000","last_fullsized":"2021-01-15T18:50:06.021154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8823,"ondisk_log_size":8823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5759142,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8969","reported_seq":"11844","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.769605+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:05.769605+0000","last_peered":"2021-01-15T18:50:05.769605+0000","last_clean":"2021-01-15T18:50:05.769605+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:05.769605+0000","last_undegraded":"2021-01-15T18:50:05.769605+0000","last_fullsized":"2021-01-15T18:50:05.769605+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8969,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6193318,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9149","reported_seq":"12160","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.099131+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:06.099131+0000","last_peered":"2021-01-15T18:50:06.099131+0000","last_clean":"2021-01-15T18:50:06.099131+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:06.099131+0000","last_undegraded":"2021-01-15T18:50:06.099131+0000","last_fullsized":"2021-01-15T18:50:06.099131+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9149,"ondisk_log_size":9149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5795923,"num_objects":708,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9047","reported_seq":"11996","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.755177+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:05.755177+0000","last_peered":"2021-01-15T18:50:05.755177+0000","last_clean":"2021-01-15T18:50:05.755177+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:05.755177+0000","last_undegraded":"2021-01-15T18:50:05.755177+0000","last_fullsized":"2021-01-15T18:50:05.755177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9047,"ondisk_log_size":9047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5582931,"num_objects":682,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9057","reported_seq":"12012","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.990479+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:05.990479+0000","last_peered":"2021-01-15T18:50:05.990479+0000","last_clean":"2021-01-15T18:50:05.990479+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:05.990479+0000","last_undegraded":"2021-01-15T18:50:05.990479+0000","last_fullsized":"2021-01-15T18:50:05.990479+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9057,"ondisk_log_size":9057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5738745,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8853","reported_seq":"11706","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.980975+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:05.980975+0000","last_peered":"2021-01-15T18:50:05.980975+0000","last_clean":"2021-01-15T18:50:05.980975+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:05.980975+0000","last_undegraded":"2021-01-15T18:50:05.980975+0000","last_fullsized":"2021-01-15T18:50:05.980975+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8853,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5742758,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8862","reported_seq":"11705","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.924806+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:05.924806+0000","last_peered":"2021-01-15T18:50:05.924806+0000","last_clean":"2021-01-15T18:50:05.924806+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:05.924806+0000","last_undegraded":"2021-01-15T18:50:05.924806+0000","last_fullsized":"2021-01-15T18:50:05.924806+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8862,"ondisk_log_size":8862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5775526,"num_objects":706,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8820","reported_seq":"11729","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:04.738778+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:04.738778+0000","last_peered":"2021-01-15T18:50:04.738778+0000","last_clean":"2021-01-15T18:50:04.738778+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:04.738778+0000","last_undegraded":"2021-01-15T18:50:04.738778+0000","last_fullsized":"2021-01-15T18:50:04.738778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8820,"ondisk_log_size":8820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5365926,"num_objects":656,"num_object_clones":0,"num_object_copies":2624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9217","reported_seq":"9457","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.104845+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:06.104845+0000","last_peered":"2021-01-15T18:50:06.104845+0000","last_clean":"2021-01-15T18:50:06.104845+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:06.104845+0000","last_undegraded":"2021-01-15T18:50:06.104845+0000","last_fullsized":"2021-01-15T18:50:06.104845+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9217,"ondisk_log_size":9217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5480614,"num_objects":670,"num_object_clones":0,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9336","reported_seq":"9979","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:05.977167+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:05.977167+0000","last_peered":"2021-01-15T18:50:05.977167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:05.977167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9336,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5783884,"num_objects":708,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8879","reported_seq":"11758","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:04.184924+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:04.184924+0000","last_peered":"2021-01-15T18:50:04.184924+0000","last_clean":"2021-01-15T18:50:04.184924+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:04.184924+0000","last_undegraded":"2021-01-15T18:50:04.184924+0000","last_fullsized":"2021-01-15T18:50:04.184924+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8879,"ondisk_log_size":8879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5406886,"num_objects":661,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91425068,"num_objects":11175,"num_object_clones":0,"num_object_copies":44700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143550,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17616,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184184832,"data_stored":184184832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143577,"ondisk_log_size":143577,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322600,"kb_used_data":45000,"kb_used_omap":0,"kb_used_meta":277567,"kb_avail":94049240,"statfs":{"total":96636764160,"available":96306421760,"internally_reserved":0,"allocated":46080000,"data_stored":45796795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":284228660},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.999},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.71499999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":1.1080000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":5.0640000000000001},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.4419999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319436,"kb_used_data":46324,"kb_used_omap":0,"kb_used_meta":273088,"kb_avail":94052404,"statfs":{"total":96636764160,"available":96309661696,"internally_reserved":0,"allocated":47435776,"data_stored":47152571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":279642112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":5.0910000000000002},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.88500000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":4.9100000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":11.189}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":16.036999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.98099999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319376,"kb_used_data":46460,"kb_used_omap":0,"kb_used_meta":272895,"kb_avail":94052464,"statfs":{"total":96636764160,"available":96309723136,"internally_reserved":0,"allocated":47575040,"data_stored":47291835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":279444532},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":5.6159999999999997},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":5.7800000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.49299999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:49:06 2021","interfaces":[{"interface":"back","average":{"1min":3.7170000000000001,"5min":4.9169999999999998,"15min":4.391},"min":{"1min":0.54900000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.391,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.1669999999999998},{"interface":"front","average":{"1min":2.3090000000000002,"5min":3.5179999999999998,"15min":3.3999999999999999},"min":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.48199999999999998},"max":{"1min":10.679,"5min":24.734999999999999,"15min":56.927},"last":1.71}]}]},{"osd":1,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312520,"kb_used_data":44972,"kb_used_omap":0,"kb_used_meta":267519,"kb_avail":94059320,"statfs":{"total":96636764160,"available":96316743680,"internally_reserved":0,"allocated":46051328,"data_stored":45768123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":273939508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.0169999999999999},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.871}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":6.2140000000000004},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":2.4540000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":2.089},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":2.1949999999999998}]}]}]}} 2021-01-15T18:50:08.296 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:09.896 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:09.897 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3791/44700 objects degraded (8.481%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":660,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11175,"data_bytes":91425068,"bytes_used":1304506368,"bytes_avail":385242550272,"bytes_total":386547056640,"degraded_objects":3791,"degraded_total":44700,"degraded_ratio":0.084809843400447421,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":159},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:09.956 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:09.957 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:11.185 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:11.195 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:11.218 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2021-01-15T18:50:10.666818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89942316,"num_objects":10994,"num_object_clones":0,"num_object_copies":43976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143731,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17616,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143758,"ondisk_log_size":143758,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1286100,"kb_used_data":175044,"kb_used_omap":2,"kb_used_meta":1110909,"kb_avail":376201260,"statfs":{"total":386547056640,"available":385230090240,"internally_reserved":0,"allocated":179245056,"data_stored":178112236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1137570972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,4,16,7,17,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17162240,"num_objects":-2095,"num_object_clones":0,"num_object_copies":-8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007702"},"pg_stats":[{"pgid":"1.f","version":"54'8782","reported_seq":"9054","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:06.017864+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:06.017864+0000","last_peered":"2021-01-15T18:50:06.017864+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:06.017864+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8782,"ondisk_log_size":8782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5947392,"num_objects":726,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8774","reported_seq":"11712","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.098919+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:06.098919+0000","last_peered":"2021-01-15T18:50:06.098919+0000","last_clean":"2021-01-15T18:50:06.098919+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:06.098919+0000","last_undegraded":"2021-01-15T18:50:06.098919+0000","last_fullsized":"2021-01-15T18:50:06.098919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8774,"ondisk_log_size":8774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5628070,"num_objects":688,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8870","reported_seq":"11815","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:09.142532+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:09.142532+0000","last_peered":"2021-01-15T18:50:09.142532+0000","last_clean":"2021-01-15T18:50:09.142532+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:09.142532+0000","last_undegraded":"2021-01-15T18:50:09.142532+0000","last_fullsized":"2021-01-15T18:50:09.142532+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8870,"ondisk_log_size":8870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5091435,"num_objects":623,"num_object_clones":0,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8869,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9088","reported_seq":"9348","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:05.874335+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:05.874335+0000","last_peered":"2021-01-15T18:50:05.874335+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:05.874335+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9088,"ondisk_log_size":9088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5767168,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9117","reported_seq":"9723","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.077247+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:06.077247+0000","last_peered":"2021-01-15T18:50:06.077247+0000","last_clean":"2021-01-15T18:50:06.077247+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:06.077247+0000","last_undegraded":"2021-01-15T18:50:06.077247+0000","last_fullsized":"2021-01-15T18:50:06.077247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9117,"ondisk_log_size":9117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5824678,"num_objects":712,"num_object_clones":0,"num_object_copies":2848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8823","reported_seq":"9048","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.021154+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:06.021154+0000","last_peered":"2021-01-15T18:50:06.021154+0000","last_clean":"2021-01-15T18:50:06.021154+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:06.021154+0000","last_undegraded":"2021-01-15T18:50:06.021154+0000","last_fullsized":"2021-01-15T18:50:06.021154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8823,"ondisk_log_size":8823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5759142,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'8969","reported_seq":"11844","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.769605+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:05.769605+0000","last_peered":"2021-01-15T18:50:05.769605+0000","last_clean":"2021-01-15T18:50:05.769605+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:05.769605+0000","last_undegraded":"2021-01-15T18:50:05.769605+0000","last_fullsized":"2021-01-15T18:50:05.769605+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8969,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6193318,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9149","reported_seq":"12160","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.099131+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:06.099131+0000","last_peered":"2021-01-15T18:50:06.099131+0000","last_clean":"2021-01-15T18:50:06.099131+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:06.099131+0000","last_undegraded":"2021-01-15T18:50:06.099131+0000","last_fullsized":"2021-01-15T18:50:06.099131+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9149,"ondisk_log_size":9149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5795923,"num_objects":708,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9047","reported_seq":"11996","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.755177+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:05.755177+0000","last_peered":"2021-01-15T18:50:05.755177+0000","last_clean":"2021-01-15T18:50:05.755177+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:05.755177+0000","last_undegraded":"2021-01-15T18:50:05.755177+0000","last_fullsized":"2021-01-15T18:50:05.755177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9047,"ondisk_log_size":9047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5582931,"num_objects":682,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9057","reported_seq":"12012","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.990479+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:05.990479+0000","last_peered":"2021-01-15T18:50:05.990479+0000","last_clean":"2021-01-15T18:50:05.990479+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:05.990479+0000","last_undegraded":"2021-01-15T18:50:05.990479+0000","last_fullsized":"2021-01-15T18:50:05.990479+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9057,"ondisk_log_size":9057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5738745,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8853","reported_seq":"11706","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.980975+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:05.980975+0000","last_peered":"2021-01-15T18:50:05.980975+0000","last_clean":"2021-01-15T18:50:05.980975+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:05.980975+0000","last_undegraded":"2021-01-15T18:50:05.980975+0000","last_fullsized":"2021-01-15T18:50:05.980975+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8853,"ondisk_log_size":8853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5742758,"num_objects":702,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8862","reported_seq":"11705","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:05.924806+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:05.924806+0000","last_peered":"2021-01-15T18:50:05.924806+0000","last_clean":"2021-01-15T18:50:05.924806+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:05.924806+0000","last_undegraded":"2021-01-15T18:50:05.924806+0000","last_fullsized":"2021-01-15T18:50:05.924806+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8862,"ondisk_log_size":8862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5775526,"num_objects":706,"num_object_clones":0,"num_object_copies":2824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8877","reported_seq":"11786","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:09.849714+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:09.849714+0000","last_peered":"2021-01-15T18:50:09.849714+0000","last_clean":"2021-01-15T18:50:09.849714+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:09.849714+0000","last_undegraded":"2021-01-15T18:50:09.849714+0000","last_fullsized":"2021-01-15T18:50:09.849714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8877,"ondisk_log_size":8877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4898982,"num_objects":599,"num_object_clones":0,"num_object_copies":2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8875,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9217","reported_seq":"9457","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:06.104845+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:06.104845+0000","last_peered":"2021-01-15T18:50:06.104845+0000","last_clean":"2021-01-15T18:50:06.104845+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:06.104845+0000","last_undegraded":"2021-01-15T18:50:06.104845+0000","last_fullsized":"2021-01-15T18:50:06.104845+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9217,"ondisk_log_size":9217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5480614,"num_objects":670,"num_object_clones":0,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9336","reported_seq":"9979","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:05.977167+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:05.977167+0000","last_peered":"2021-01-15T18:50:05.977167+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:05.977167+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9336,"ondisk_log_size":9336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5783884,"num_objects":708,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9333,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8937","reported_seq":"11816","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:09.093593+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:09.093593+0000","last_peered":"2021-01-15T18:50:09.093593+0000","last_clean":"2021-01-15T18:50:09.093593+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:09.093593+0000","last_undegraded":"2021-01-15T18:50:09.093593+0000","last_fullsized":"2021-01-15T18:50:09.093593+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8937,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4931750,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8935,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":89942316,"num_objects":10994,"num_object_clones":0,"num_object_copies":43976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143731,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17616,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176500736,"data_stored":176500736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143758,"ondisk_log_size":143758,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215231,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322600,"kb_used_data":45000,"kb_used_omap":0,"kb_used_meta":277567,"kb_avail":94049240,"statfs":{"total":96636764160,"available":96306421760,"internally_reserved":0,"allocated":46080000,"data_stored":45796795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":284228660},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":0.999},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.71499999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":1.1080000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":5.0640000000000001},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":1.4419999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325488,"kb_used_data":42456,"kb_used_omap":0,"kb_used_meta":283008,"kb_avail":94046352,"statfs":{"total":96636764160,"available":96303464448,"internally_reserved":0,"allocated":43474944,"data_stored":43191739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":289800192},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":3.2389999999999999},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":3.036},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":0.496}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":2.1139999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":9.407}]}]},{"osd":0,"up_from":22,"seq":94489280645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325492,"kb_used_data":42616,"kb_used_omap":0,"kb_used_meta":282815,"kb_avail":94046348,"statfs":{"total":96636764160,"available":96303460352,"internally_reserved":0,"allocated":43638784,"data_stored":43355579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":289602612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,2,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":6.7060000000000004},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":6.8120000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":3.0009999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.70299999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.05}]}]},{"osd":1,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312520,"kb_used_data":44972,"kb_used_omap":0,"kb_used_meta":267519,"kb_avail":94059320,"statfs":{"total":96636764160,"available":96316743680,"internally_reserved":0,"allocated":46051328,"data_stored":45768123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":273939508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.0169999999999999},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.871}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":6.2140000000000004},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":2.4540000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":2.089},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":2.1949999999999998}]}]}]}} 2021-01-15T18:50:14.221 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:15.225 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:15.227 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:15.239 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2021-01-15T18:50:14.667646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82684204,"num_objects":10108,"num_object_clones":0,"num_object_copies":40432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144617,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17659,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144644,"ondisk_log_size":144644,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1276308,"kb_used_data":164380,"kb_used_omap":2,"kb_used_meta":1111677,"kb_avail":376211052,"statfs":{"total":386547056640,"available":385240117248,"internally_reserved":0,"allocated":168325120,"data_stored":167192300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1138357404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,30,7,26,22,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":141,"apply_latency_ms":141,"commit_latency_ns":141000000,"apply_latency_ns":141000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16318464,"num_objects":-1992,"num_object_clones":0,"num_object_copies":-7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007685"},"pg_stats":[{"pgid":"1.f","version":"54'8844","reported_seq":"9115","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:10.940771+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:10.940771+0000","last_peered":"2021-01-15T18:50:10.940771+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:10.940771+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8844,"ondisk_log_size":8844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5439488,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8830","reported_seq":"11767","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.164009+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:11.164009+0000","last_peered":"2021-01-15T18:50:11.164009+0000","last_clean":"2021-01-15T18:50:11.164009+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:11.164009+0000","last_undegraded":"2021-01-15T18:50:11.164009+0000","last_fullsized":"2021-01-15T18:50:11.164009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8830,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5169318,"num_objects":632,"num_object_clones":0,"num_object_copies":2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8925","reported_seq":"11870","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:14.256300+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:14.256300+0000","last_peered":"2021-01-15T18:50:14.256300+0000","last_clean":"2021-01-15T18:50:14.256300+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:14.256300+0000","last_undegraded":"2021-01-15T18:50:14.256300+0000","last_fullsized":"2021-01-15T18:50:14.256300+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8925,"ondisk_log_size":8925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4640875,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8924,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9162","reported_seq":"9422","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:11.000275+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:11.000275+0000","last_peered":"2021-01-15T18:50:11.000275+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:11.000275+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9162,"ondisk_log_size":9162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5160960,"num_objects":630,"num_object_clones":0,"num_object_copies":2520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9178","reported_seq":"9784","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.152570+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:11.152570+0000","last_peered":"2021-01-15T18:50:11.152570+0000","last_clean":"2021-01-15T18:50:11.152570+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:11.152570+0000","last_undegraded":"2021-01-15T18:50:11.152570+0000","last_fullsized":"2021-01-15T18:50:11.152570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9178,"ondisk_log_size":9178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5324966,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8884","reported_seq":"9109","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.148550+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:11.148550+0000","last_peered":"2021-01-15T18:50:11.148550+0000","last_clean":"2021-01-15T18:50:11.148550+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:11.148550+0000","last_undegraded":"2021-01-15T18:50:11.148550+0000","last_fullsized":"2021-01-15T18:50:11.148550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8884,"ondisk_log_size":8884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5259430,"num_objects":643,"num_object_clones":0,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9019","reported_seq":"11894","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.148629+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:11.148629+0000","last_peered":"2021-01-15T18:50:11.148629+0000","last_clean":"2021-01-15T18:50:11.148629+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:11.148629+0000","last_undegraded":"2021-01-15T18:50:11.148629+0000","last_fullsized":"2021-01-15T18:50:11.148629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9019,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5783718,"num_objects":707,"num_object_clones":0,"num_object_copies":2828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9205","reported_seq":"12216","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.224736+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:11.224736+0000","last_peered":"2021-01-15T18:50:11.224736+0000","last_clean":"2021-01-15T18:50:11.224736+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:11.224736+0000","last_undegraded":"2021-01-15T18:50:11.224736+0000","last_fullsized":"2021-01-15T18:50:11.224736+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9205,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5337171,"num_objects":652,"num_object_clones":0,"num_object_copies":2608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9204,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9102","reported_seq":"12049","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.121880+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:11.121880+0000","last_peered":"2021-01-15T18:50:11.121880+0000","last_clean":"2021-01-15T18:50:11.121880+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:11.121880+0000","last_undegraded":"2021-01-15T18:50:11.121880+0000","last_fullsized":"2021-01-15T18:50:11.121880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9102,"ondisk_log_size":9102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5132371,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9101,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9131","reported_seq":"12085","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:10.982063+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:10.982063+0000","last_peered":"2021-01-15T18:50:10.982063+0000","last_clean":"2021-01-15T18:50:10.982063+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:10.982063+0000","last_undegraded":"2021-01-15T18:50:10.982063+0000","last_fullsized":"2021-01-15T18:50:10.982063+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9131,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5132537,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8904","reported_seq":"11757","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:10.600969+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:10.600969+0000","last_peered":"2021-01-15T18:50:10.600969+0000","last_clean":"2021-01-15T18:50:10.600969+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:10.600969+0000","last_undegraded":"2021-01-15T18:50:10.600969+0000","last_fullsized":"2021-01-15T18:50:10.600969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8904,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5324966,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8902,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8924","reported_seq":"11767","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.084874+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:11.084874+0000","last_peered":"2021-01-15T18:50:11.084874+0000","last_clean":"2021-01-15T18:50:11.084874+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:11.084874+0000","last_undegraded":"2021-01-15T18:50:11.084874+0000","last_fullsized":"2021-01-15T18:50:11.084874+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8924,"ondisk_log_size":8924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5267622,"num_objects":644,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8877","reported_seq":"11786","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:09.849714+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:09.849714+0000","last_peered":"2021-01-15T18:50:09.849714+0000","last_clean":"2021-01-15T18:50:09.849714+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:09.849714+0000","last_undegraded":"2021-01-15T18:50:09.849714+0000","last_fullsized":"2021-01-15T18:50:09.849714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8877,"ondisk_log_size":8877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4898982,"num_objects":599,"num_object_clones":0,"num_object_copies":2396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8875,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9266","reported_seq":"9505","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:11.221071+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:11.221071+0000","last_peered":"2021-01-15T18:50:11.221071+0000","last_clean":"2021-01-15T18:50:11.221071+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:11.221071+0000","last_undegraded":"2021-01-15T18:50:11.221071+0000","last_fullsized":"2021-01-15T18:50:11.221071+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9266,"ondisk_log_size":9266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5079206,"num_objects":621,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9264,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9401","reported_seq":"10130","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:11.007691+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:11.007691+0000","last_peered":"2021-01-15T18:50:11.007691+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:11.007691+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9401,"ondisk_log_size":9401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5251404,"num_objects":643,"num_object_clones":0,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-521}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8992","reported_seq":"11871","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:14.012496+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:14.012496+0000","last_peered":"2021-01-15T18:50:14.012496+0000","last_clean":"2021-01-15T18:50:14.012496+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:14.012496+0000","last_undegraded":"2021-01-15T18:50:14.012496+0000","last_fullsized":"2021-01-15T18:50:14.012496+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8992,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4481190,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":82684204,"num_objects":10108,"num_object_clones":0,"num_object_copies":40432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144617,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17659,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164868096,"data_stored":164868096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144644,"ondisk_log_size":144644,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215232,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324724,"kb_used_data":41308,"kb_used_omap":0,"kb_used_meta":283327,"kb_avail":94047116,"statfs":{"total":96636764160,"available":96304246784,"internally_reserved":0,"allocated":42299392,"data_stored":42016187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":290126900},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,18,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":10.239000000000001},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":9.7629999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.86599999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.385},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":0.75700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311480,"kb_used_data":39284,"kb_used_omap":0,"kb_used_meta":272128,"kb_avail":94060360,"statfs":{"total":96636764160,"available":96317808640,"internally_reserved":0,"allocated":40226816,"data_stored":39943611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":278659072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":4.1539999999999999},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.5820000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":8.1319999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.621}]}]},{"osd":0,"up_from":22,"seq":94489280645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325492,"kb_used_data":42616,"kb_used_omap":0,"kb_used_meta":282815,"kb_avail":94046348,"statfs":{"total":96636764160,"available":96303460352,"internally_reserved":0,"allocated":43638784,"data_stored":43355579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":289602612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,2,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":6.7060000000000004},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":6.8120000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":3.0009999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.70299999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.05}]}]},{"osd":1,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314612,"kb_used_data":41172,"kb_used_omap":0,"kb_used_meta":273407,"kb_avail":94057228,"statfs":{"total":96636764160,"available":96314601472,"internally_reserved":0,"allocated":42160128,"data_stored":41876923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":279968820},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,20,0,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":38.216000000000001},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.0009999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":11.808999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":4.0750000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.175}]}]}]}} 2021-01-15T18:50:15.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:16.370 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3748/40432 objects degraded (9.270%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":666,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10108,"data_bytes":82684204,"bytes_used":1306939392,"bytes_avail":385240117248,"bytes_total":386547056640,"degraded_objects":3748,"degraded_total":40432,"degraded_ratio":0.092698852394143258,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":165},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:16.417 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:16.417 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:17.619 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:17.621 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:17.635 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2021-01-15T18:50:16.669807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75917612,"num_objects":9282,"num_object_clones":0,"num_object_copies":37128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145443,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17710,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145470,"ondisk_log_size":145470,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1257764,"kb_used_data":152864,"kb_used_omap":2,"kb_used_meta":1104701,"kb_avail":376229596,"statfs":{"total":386547056640,"available":385259106304,"internally_reserved":0,"allocated":156532736,"data_stored":155399916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1131213980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,6,8,12,10,18,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":139,"apply_latency_ms":139,"commit_latency_ns":139000000,"apply_latency_ns":139000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22331392,"num_objects":-2726,"num_object_clones":0,"num_object_copies":-10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009409"},"pg_stats":[{"pgid":"1.f","version":"54'8904","reported_seq":"9176","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:15.790647+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:15.790647+0000","last_peered":"2021-01-15T18:50:15.790647+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:15.790647+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8904,"ondisk_log_size":8904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4947968,"num_objects":604,"num_object_clones":0,"num_object_copies":2416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8904,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8884","reported_seq":"11822","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.030208+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:16.030208+0000","last_peered":"2021-01-15T18:50:16.030208+0000","last_clean":"2021-01-15T18:50:16.030208+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:16.030208+0000","last_undegraded":"2021-01-15T18:50:16.030208+0000","last_fullsized":"2021-01-15T18:50:16.030208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8884,"ondisk_log_size":8884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4726950,"num_objects":578,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8925","reported_seq":"11870","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:14.256300+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:14.256300+0000","last_peered":"2021-01-15T18:50:14.256300+0000","last_clean":"2021-01-15T18:50:14.256300+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:14.256300+0000","last_undegraded":"2021-01-15T18:50:14.256300+0000","last_fullsized":"2021-01-15T18:50:14.256300+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8925,"ondisk_log_size":8925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4640875,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8924,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9213","reported_seq":"9473","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:15.959735+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:15.959735+0000","last_peered":"2021-01-15T18:50:15.959735+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:15.959735+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9213,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4743168,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9238","reported_seq":"9843","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.302781+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:16.302781+0000","last_peered":"2021-01-15T18:50:16.302781+0000","last_clean":"2021-01-15T18:50:16.302781+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:16.302781+0000","last_undegraded":"2021-01-15T18:50:16.302781+0000","last_fullsized":"2021-01-15T18:50:16.302781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9238,"ondisk_log_size":9238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4833446,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9236,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8948","reported_seq":"9173","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.246647+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:16.246647+0000","last_peered":"2021-01-15T18:50:16.246647+0000","last_clean":"2021-01-15T18:50:16.246647+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:16.246647+0000","last_undegraded":"2021-01-15T18:50:16.246647+0000","last_fullsized":"2021-01-15T18:50:16.246647+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8948,"ondisk_log_size":8948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4735142,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8946,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9081","reported_seq":"11955","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.284318+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:16.284318+0000","last_peered":"2021-01-15T18:50:16.284318+0000","last_clean":"2021-01-15T18:50:16.284318+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:16.284318+0000","last_undegraded":"2021-01-15T18:50:16.284318+0000","last_fullsized":"2021-01-15T18:50:16.284318+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9081,"ondisk_log_size":9081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5275814,"num_objects":645,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9079,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9259","reported_seq":"12270","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.059507+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:16.059507+0000","last_peered":"2021-01-15T18:50:16.059507+0000","last_clean":"2021-01-15T18:50:16.059507+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:16.059507+0000","last_undegraded":"2021-01-15T18:50:16.059507+0000","last_fullsized":"2021-01-15T18:50:16.059507+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9259,"ondisk_log_size":9259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4894803,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9167","reported_seq":"12116","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.087700+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:16.087700+0000","last_peered":"2021-01-15T18:50:16.087700+0000","last_clean":"2021-01-15T18:50:16.087700+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:16.087700+0000","last_undegraded":"2021-01-15T18:50:16.087700+0000","last_fullsized":"2021-01-15T18:50:16.087700+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9167,"ondisk_log_size":9167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4599891,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9166,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9186","reported_seq":"12138","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.088245+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:16.088245+0000","last_peered":"2021-01-15T18:50:16.088245+0000","last_clean":"2021-01-15T18:50:16.088245+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:16.088245+0000","last_undegraded":"2021-01-15T18:50:16.088245+0000","last_fullsized":"2021-01-15T18:50:16.088245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9186,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4681977,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8968","reported_seq":"11821","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:15.940129+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:15.940129+0000","last_peered":"2021-01-15T18:50:15.940129+0000","last_clean":"2021-01-15T18:50:15.940129+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:15.940129+0000","last_undegraded":"2021-01-15T18:50:15.940129+0000","last_fullsized":"2021-01-15T18:50:15.940129+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8968,"ondisk_log_size":8968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4800678,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8981","reported_seq":"11824","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.088391+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:16.088391+0000","last_peered":"2021-01-15T18:50:16.088391+0000","last_clean":"2021-01-15T18:50:16.088391+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:16.088391+0000","last_undegraded":"2021-01-15T18:50:16.088391+0000","last_fullsized":"2021-01-15T18:50:16.088391+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8981,"ondisk_log_size":8981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4800678,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8979,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8941","reported_seq":"11848","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:14.922158+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:14.922158+0000","last_peered":"2021-01-15T18:50:14.922158+0000","last_clean":"2021-01-15T18:50:14.922158+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:14.922158+0000","last_undegraded":"2021-01-15T18:50:14.922158+0000","last_fullsized":"2021-01-15T18:50:14.922158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8941,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4374694,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8939,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9323","reported_seq":"9563","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.030459+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:16.030459+0000","last_peered":"2021-01-15T18:50:16.030459+0000","last_clean":"2021-01-15T18:50:16.030459+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:16.030459+0000","last_undegraded":"2021-01-15T18:50:16.030459+0000","last_fullsized":"2021-01-15T18:50:16.030459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9323,"ondisk_log_size":9323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4612262,"num_objects":564,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9460","reported_seq":"10291","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:16.072059+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:16.072059+0000","last_peered":"2021-01-15T18:50:16.072059+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:16.072059+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4768076,"num_objects":584,"num_object_clones":0,"num_object_copies":2336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'8992","reported_seq":"11871","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:14.012496+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:14.012496+0000","last_peered":"2021-01-15T18:50:14.012496+0000","last_clean":"2021-01-15T18:50:14.012496+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:14.012496+0000","last_undegraded":"2021-01-15T18:50:14.012496+0000","last_fullsized":"2021-01-15T18:50:14.012496+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8992,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4481190,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":75917612,"num_objects":9282,"num_object_clones":0,"num_object_copies":37128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145443,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17710,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":153112576,"data_stored":153112576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145470,"ondisk_log_size":145470,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314380,"kb_used_data":37476,"kb_used_omap":0,"kb_used_meta":276863,"kb_avail":94057460,"statfs":{"total":96636764160,"available":96314839040,"internally_reserved":0,"allocated":38375424,"data_stored":38092219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":283507764},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,1,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":3.4329999999999998},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.81100000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":3.9359999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":4.6529999999999996},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.6160000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311480,"kb_used_data":39284,"kb_used_omap":0,"kb_used_meta":272128,"kb_avail":94060360,"statfs":{"total":96636764160,"available":96317808640,"internally_reserved":0,"allocated":40226816,"data_stored":39943611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":278659072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,1,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":4.1539999999999999},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.52500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.5820000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":8.1319999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.621}]}]},{"osd":0,"up_from":22,"seq":94489280646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311088,"kb_used_data":38744,"kb_used_omap":0,"kb_used_meta":272319,"kb_avail":94060752,"statfs":{"total":96636764160,"available":96318210048,"internally_reserved":0,"allocated":39673856,"data_stored":39390651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":278854708},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":8.7249999999999996},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":8.8140000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":4.8869999999999996},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":1.5760000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.3079999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.6679999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320816,"kb_used_data":37360,"kb_used_omap":0,"kb_used_meta":283391,"kb_avail":94051024,"statfs":{"total":96636764160,"available":96308248576,"internally_reserved":0,"allocated":38256640,"data_stored":37973435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":290192436},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":13.236000000000001},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":6.8540000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":27.969999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":6.9489999999999998},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":7.0350000000000001}]}]}]}} 2021-01-15T18:50:20.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:21.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:21.846 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:21.921 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2021-01-15T18:50:20.675030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74565932,"num_objects":9117,"num_object_clones":0,"num_object_copies":36468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145608,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17710,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145635,"ondisk_log_size":145635,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1270160,"kb_used_data":145624,"kb_used_omap":2,"kb_used_meta":1124349,"kb_avail":376217200,"statfs":{"total":386547056640,"available":385246412800,"internally_reserved":0,"allocated":149118976,"data_stored":147986156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1151333532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,4,16,9,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":133,"apply_latency_ms":133,"commit_latency_ns":133000000,"apply_latency_ns":133000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16859136,"num_objects":-2058,"num_object_clones":0,"num_object_copies":-8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010205"},"pg_stats":[{"pgid":"1.f","version":"54'8904","reported_seq":"9176","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:15.790647+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:15.790647+0000","last_peered":"2021-01-15T18:50:15.790647+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:15.790647+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8904,"ondisk_log_size":8904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4947968,"num_objects":604,"num_object_clones":0,"num_object_copies":2416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8904,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8884","reported_seq":"11822","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.030208+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:16.030208+0000","last_peered":"2021-01-15T18:50:16.030208+0000","last_clean":"2021-01-15T18:50:16.030208+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:16.030208+0000","last_undegraded":"2021-01-15T18:50:16.030208+0000","last_fullsized":"2021-01-15T18:50:16.030208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8884,"ondisk_log_size":8884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4726950,"num_objects":578,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8979","reported_seq":"11924","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.183437+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:19.183437+0000","last_peered":"2021-01-15T18:50:19.183437+0000","last_clean":"2021-01-15T18:50:19.183437+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:19.183437+0000","last_undegraded":"2021-01-15T18:50:19.183437+0000","last_fullsized":"2021-01-15T18:50:19.183437+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8979,"ondisk_log_size":8979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4198507,"num_objects":514,"num_object_clones":0,"num_object_copies":2056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8978,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9213","reported_seq":"9473","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:15.959735+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:15.959735+0000","last_peered":"2021-01-15T18:50:15.959735+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:15.959735+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9213,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4743168,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9238","reported_seq":"9843","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.302781+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:16.302781+0000","last_peered":"2021-01-15T18:50:16.302781+0000","last_clean":"2021-01-15T18:50:16.302781+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:16.302781+0000","last_undegraded":"2021-01-15T18:50:16.302781+0000","last_fullsized":"2021-01-15T18:50:16.302781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9238,"ondisk_log_size":9238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4833446,"num_objects":591,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9236,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'8948","reported_seq":"9173","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.246647+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:16.246647+0000","last_peered":"2021-01-15T18:50:16.246647+0000","last_clean":"2021-01-15T18:50:16.246647+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:16.246647+0000","last_undegraded":"2021-01-15T18:50:16.246647+0000","last_fullsized":"2021-01-15T18:50:16.246647+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8948,"ondisk_log_size":8948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4735142,"num_objects":579,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8946,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9081","reported_seq":"11955","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.284318+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:16.284318+0000","last_peered":"2021-01-15T18:50:16.284318+0000","last_clean":"2021-01-15T18:50:16.284318+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:16.284318+0000","last_undegraded":"2021-01-15T18:50:16.284318+0000","last_fullsized":"2021-01-15T18:50:16.284318+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9081,"ondisk_log_size":9081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5275814,"num_objects":645,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9079,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9259","reported_seq":"12270","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.059507+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:16.059507+0000","last_peered":"2021-01-15T18:50:16.059507+0000","last_clean":"2021-01-15T18:50:16.059507+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:16.059507+0000","last_undegraded":"2021-01-15T18:50:16.059507+0000","last_fullsized":"2021-01-15T18:50:16.059507+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9259,"ondisk_log_size":9259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4894803,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9167","reported_seq":"12116","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.087700+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:16.087700+0000","last_peered":"2021-01-15T18:50:16.087700+0000","last_clean":"2021-01-15T18:50:16.087700+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:16.087700+0000","last_undegraded":"2021-01-15T18:50:16.087700+0000","last_fullsized":"2021-01-15T18:50:16.087700+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9167,"ondisk_log_size":9167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4599891,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9166,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9186","reported_seq":"12138","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.088245+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:16.088245+0000","last_peered":"2021-01-15T18:50:16.088245+0000","last_clean":"2021-01-15T18:50:16.088245+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:16.088245+0000","last_undegraded":"2021-01-15T18:50:16.088245+0000","last_fullsized":"2021-01-15T18:50:16.088245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9186,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4681977,"num_objects":573,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'8968","reported_seq":"11821","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:15.940129+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:15.940129+0000","last_peered":"2021-01-15T18:50:15.940129+0000","last_clean":"2021-01-15T18:50:15.940129+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:15.940129+0000","last_undegraded":"2021-01-15T18:50:15.940129+0000","last_fullsized":"2021-01-15T18:50:15.940129+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8968,"ondisk_log_size":8968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4800678,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'8981","reported_seq":"11824","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.088391+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:16.088391+0000","last_peered":"2021-01-15T18:50:16.088391+0000","last_clean":"2021-01-15T18:50:16.088391+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:16.088391+0000","last_undegraded":"2021-01-15T18:50:16.088391+0000","last_fullsized":"2021-01-15T18:50:16.088391+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8981,"ondisk_log_size":8981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4800678,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8979,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8994","reported_seq":"11903","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.925570+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:19.925570+0000","last_peered":"2021-01-15T18:50:19.925570+0000","last_clean":"2021-01-15T18:50:19.925570+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:19.925570+0000","last_undegraded":"2021-01-15T18:50:19.925570+0000","last_fullsized":"2021-01-15T18:50:19.925570+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8994,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3940518,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9323","reported_seq":"9563","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:16.030459+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:16.030459+0000","last_peered":"2021-01-15T18:50:16.030459+0000","last_clean":"2021-01-15T18:50:16.030459+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:16.030459+0000","last_undegraded":"2021-01-15T18:50:16.030459+0000","last_fullsized":"2021-01-15T18:50:16.030459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9323,"ondisk_log_size":9323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4612262,"num_objects":564,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9460","reported_seq":"10291","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:16.072059+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:16.072059+0000","last_peered":"2021-01-15T18:50:16.072059+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:16.072059+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4768076,"num_objects":584,"num_object_clones":0,"num_object_copies":2336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9050","reported_seq":"11926","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.246910+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:19.246910+0000","last_peered":"2021-01-15T18:50:19.246910+0000","last_clean":"2021-01-15T18:50:19.246910+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:19.246910+0000","last_undegraded":"2021-01-15T18:50:19.246910+0000","last_fullsized":"2021-01-15T18:50:19.246910+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9050,"ondisk_log_size":9050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4006054,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74565932,"num_objects":9117,"num_object_clones":0,"num_object_copies":36468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145608,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17710,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145952768,"data_stored":145952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145635,"ondisk_log_size":145635,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215233,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314380,"kb_used_data":37476,"kb_used_omap":0,"kb_used_meta":276863,"kb_avail":94057460,"statfs":{"total":96636764160,"available":96314839040,"internally_reserved":0,"allocated":38375424,"data_stored":38092219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":283507764},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,1,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":3.234,"15min":3.2069999999999999},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.5530000000000008,"5min":24.763000000000002,"15min":24.763000000000002},"last":3.4329999999999998},{"interface":"front","average":{"1min":2.101,"5min":2.532,"15min":3.1160000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.3759999999999994,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.81100000000000005}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":3.9359999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:12 2021","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":4.1529999999999996,"15min":3.069},"min":{"1min":0.48699999999999999,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":9.2530000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":4.6529999999999996},{"interface":"front","average":{"1min":5.8090000000000002,"5min":4.4029999999999996,"15min":3.6619999999999999},"min":{"1min":0.60999999999999999,"5min":0.56899999999999995,"15min":0.503},"max":{"1min":22.146000000000001,"5min":24.103000000000002,"15min":24.103000000000002},"last":3.6160000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317680,"kb_used_data":35648,"kb_used_omap":0,"kb_used_meta":281984,"kb_avail":94054160,"statfs":{"total":96636764160,"available":96311459840,"internally_reserved":0,"allocated":36503552,"data_stored":36220347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":288751616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":2.4060000000000001},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.6479999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.256},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":1.4330000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":19.338999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":1.7430000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317284,"kb_used_data":35140,"kb_used_omap":0,"kb_used_meta":282111,"kb_avail":94054556,"statfs":{"total":96636764160,"available":96311865344,"internally_reserved":0,"allocated":35983360,"data_stored":35700155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":288881716},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.39100000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.86099999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.0529999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320816,"kb_used_data":37360,"kb_used_omap":0,"kb_used_meta":283391,"kb_avail":94051024,"statfs":{"total":96636764160,"available":96308248576,"internally_reserved":0,"allocated":38256640,"data_stored":37973435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":290192436},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,8,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":13.236000000000001},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":6.8540000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":27.969999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":6.9489999999999998},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":7.0350000000000001}]}]}]}} 2021-01-15T18:50:21.923 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:23.212 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:23.213 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3625/33680 objects degraded (10.763%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":673,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8420,"data_bytes":68856108,"bytes_used":1292779520,"bytes_avail":385254277120,"bytes_total":386547056640,"degraded_objects":3625,"degraded_total":33680,"degraded_ratio":0.10763064133016627,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:23.238 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:23.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:24.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:24.133 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:24.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2021-01-15T18:50:22.676024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68856108,"num_objects":8420,"num_object_clones":0,"num_object_copies":33680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146305,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17782,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146332,"ondisk_log_size":146332,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1262480,"kb_used_data":138812,"kb_used_omap":2,"kb_used_meta":1123517,"kb_avail":376224880,"statfs":{"total":386547056640,"available":385254277120,"internally_reserved":0,"allocated":142143488,"data_stored":141010668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1150481564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,12,17,12,22,5,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21086208,"num_objects":-2574,"num_object_clones":0,"num_object_copies":-10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009206"},"pg_stats":[{"pgid":"1.f","version":"54'8964","reported_seq":"9235","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:21.098596+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:21.098596+0000","last_peered":"2021-01-15T18:50:21.098596+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:21.098596+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8964,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4456448,"num_objects":544,"num_object_clones":0,"num_object_copies":2176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8934","reported_seq":"11872","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.115146+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:21.115146+0000","last_peered":"2021-01-15T18:50:21.115146+0000","last_clean":"2021-01-15T18:50:21.115146+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:21.115146+0000","last_undegraded":"2021-01-15T18:50:21.115146+0000","last_fullsized":"2021-01-15T18:50:21.115146+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8934,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4317350,"num_objects":528,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'8979","reported_seq":"11924","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.183437+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:19.183437+0000","last_peered":"2021-01-15T18:50:19.183437+0000","last_clean":"2021-01-15T18:50:19.183437+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:19.183437+0000","last_undegraded":"2021-01-15T18:50:19.183437+0000","last_fullsized":"2021-01-15T18:50:19.183437+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8979,"ondisk_log_size":8979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4198507,"num_objects":514,"num_object_clones":0,"num_object_copies":2056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8978,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9279","reported_seq":"9538","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:21.105909+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:21.105909+0000","last_peered":"2021-01-15T18:50:21.105909+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:21.105909+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9279,"ondisk_log_size":9279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4202496,"num_objects":513,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9288","reported_seq":"9894","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.225563+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:21.225563+0000","last_peered":"2021-01-15T18:50:21.225563+0000","last_clean":"2021-01-15T18:50:21.225563+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:21.225563+0000","last_undegraded":"2021-01-15T18:50:21.225563+0000","last_fullsized":"2021-01-15T18:50:21.225563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9288,"ondisk_log_size":9288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4423846,"num_objects":541,"num_object_clones":0,"num_object_copies":2164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9015","reported_seq":"9240","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.009126+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:21.009126+0000","last_peered":"2021-01-15T18:50:21.009126+0000","last_clean":"2021-01-15T18:50:21.009126+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:21.009126+0000","last_undegraded":"2021-01-15T18:50:21.009126+0000","last_fullsized":"2021-01-15T18:50:21.009126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9015,"ondisk_log_size":9015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4186278,"num_objects":512,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9139","reported_seq":"12013","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.239291+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:21.239291+0000","last_peered":"2021-01-15T18:50:21.239291+0000","last_clean":"2021-01-15T18:50:21.239291+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:21.239291+0000","last_undegraded":"2021-01-15T18:50:21.239291+0000","last_fullsized":"2021-01-15T18:50:21.239291+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9139,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4800678,"num_objects":587,"num_object_clones":0,"num_object_copies":2348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9309","reported_seq":"12320","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.206335+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:21.206335+0000","last_peered":"2021-01-15T18:50:21.206335+0000","last_clean":"2021-01-15T18:50:21.206335+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:21.206335+0000","last_undegraded":"2021-01-15T18:50:21.206335+0000","last_fullsized":"2021-01-15T18:50:21.206335+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9309,"ondisk_log_size":9309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4485203,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9215","reported_seq":"12164","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.236006+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:21.236006+0000","last_peered":"2021-01-15T18:50:21.236006+0000","last_clean":"2021-01-15T18:50:21.236006+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:21.236006+0000","last_undegraded":"2021-01-15T18:50:21.236006+0000","last_fullsized":"2021-01-15T18:50:21.236006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9215,"ondisk_log_size":9215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4206675,"num_objects":514,"num_object_clones":0,"num_object_copies":2056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9238","reported_seq":"12193","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:20.828783+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:20.828783+0000","last_peered":"2021-01-15T18:50:20.828783+0000","last_clean":"2021-01-15T18:50:20.828783+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:20.828783+0000","last_undegraded":"2021-01-15T18:50:20.828783+0000","last_fullsized":"2021-01-15T18:50:20.828783+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9238,"ondisk_log_size":9238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4255993,"num_objects":521,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9235,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9022","reported_seq":"11875","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.051894+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:21.051894+0000","last_peered":"2021-01-15T18:50:21.051894+0000","last_clean":"2021-01-15T18:50:21.051894+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:21.051894+0000","last_undegraded":"2021-01-15T18:50:21.051894+0000","last_fullsized":"2021-01-15T18:50:21.051894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9022,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4358310,"num_objects":533,"num_object_clones":0,"num_object_copies":2132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9020","reported_seq":"11863","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.146335+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:21.146335+0000","last_peered":"2021-01-15T18:50:21.146335+0000","last_clean":"2021-01-15T18:50:21.146335+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:21.146335+0000","last_undegraded":"2021-01-15T18:50:21.146335+0000","last_fullsized":"2021-01-15T18:50:21.146335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9020,"ondisk_log_size":9020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4481190,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'8994","reported_seq":"11903","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.925570+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:19.925570+0000","last_peered":"2021-01-15T18:50:19.925570+0000","last_clean":"2021-01-15T18:50:19.925570+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:19.925570+0000","last_undegraded":"2021-01-15T18:50:19.925570+0000","last_fullsized":"2021-01-15T18:50:19.925570+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8994,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3940518,"num_objects":482,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9366","reported_seq":"9606","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:21.109004+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:21.109004+0000","last_peered":"2021-01-15T18:50:21.109004+0000","last_clean":"2021-01-15T18:50:21.109004+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:21.109004+0000","last_undegraded":"2021-01-15T18:50:21.109004+0000","last_fullsized":"2021-01-15T18:50:21.109004+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9366,"ondisk_log_size":9366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4260006,"num_objects":521,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9520","reported_seq":"10495","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:21.113373+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:21.113373+0000","last_peered":"2021-01-15T18:50:21.113373+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:21.113373+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9520,"ondisk_log_size":9520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4276556,"num_objects":524,"num_object_clones":0,"num_object_copies":2096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9517,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9050","reported_seq":"11926","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:19.246910+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:19.246910+0000","last_peered":"2021-01-15T18:50:19.246910+0000","last_clean":"2021-01-15T18:50:19.246910+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:19.246910+0000","last_undegraded":"2021-01-15T18:50:19.246910+0000","last_fullsized":"2021-01-15T18:50:19.246910+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9050,"ondisk_log_size":9050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4006054,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":68856108,"num_objects":8420,"num_object_clones":0,"num_object_copies":33680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146305,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17782,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138977280,"data_stored":138977280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146332,"ondisk_log_size":146332,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215234,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316660,"kb_used_data":34080,"kb_used_omap":0,"kb_used_meta":282559,"kb_avail":94055180,"statfs":{"total":96636764160,"available":96312504320,"internally_reserved":0,"allocated":34897920,"data_stored":34614715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":289340468},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,11,2,6,3,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":1.5229999999999999},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":2.2450000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.745},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.872}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317680,"kb_used_data":35648,"kb_used_omap":0,"kb_used_meta":281984,"kb_avail":94054160,"statfs":{"total":96636764160,"available":96311459840,"internally_reserved":0,"allocated":36503552,"data_stored":36220347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":288751616},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":2.4060000000000001},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":1.6479999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.256},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":1.4330000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":19.338999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":1.7430000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317284,"kb_used_data":35140,"kb_used_omap":0,"kb_used_meta":282111,"kb_avail":94054556,"statfs":{"total":96636764160,"available":96311865344,"internally_reserved":0,"allocated":35983360,"data_stored":35700155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":288881716},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.39100000000000001},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.86099999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.96099999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.0529999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310856,"kb_used_data":33944,"kb_used_omap":0,"kb_used_meta":276863,"kb_avail":94060984,"statfs":{"total":96636764160,"available":96318447616,"internally_reserved":0,"allocated":34758656,"data_stored":34475451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":283507764},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":6.5099999999999998},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":3.9249999999999998},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":4.5419999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":4.3010000000000002},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":4.4249999999999998}]}]}]}} 2021-01-15T18:50:27.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:28.397 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:28.437 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2021-01-15T18:50:26.676871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":61163820,"num_objects":7481,"num_object_clones":0,"num_object_copies":29924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147244,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17817,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147271,"ondisk_log_size":147271,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1250436,"kb_used_data":124252,"kb_used_omap":2,"kb_used_meta":1126077,"kb_avail":376236924,"statfs":{"total":386547056640,"available":385266610176,"internally_reserved":0,"allocated":127234048,"data_stored":126101228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1153103004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,6,7,5,9,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21520384,"num_objects":-2627,"num_object_clones":0,"num_object_copies":-10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009224"},"pg_stats":[{"pgid":"1.f","version":"54'9013","reported_seq":"9285","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:25.994931+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:25.994931+0000","last_peered":"2021-01-15T18:50:25.994931+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:25.994931+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9013,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4055040,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8987","reported_seq":"11925","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.332272+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:26.332272+0000","last_peered":"2021-01-15T18:50:26.332272+0000","last_clean":"2021-01-15T18:50:26.332272+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:26.332272+0000","last_undegraded":"2021-01-15T18:50:26.332272+0000","last_fullsized":"2021-01-15T18:50:26.332272+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8987,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3883174,"num_objects":475,"num_object_clones":0,"num_object_copies":1900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9033","reported_seq":"11978","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:24.314258+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:24.314258+0000","last_peered":"2021-01-15T18:50:24.314258+0000","last_clean":"2021-01-15T18:50:24.314258+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:24.314258+0000","last_undegraded":"2021-01-15T18:50:24.314258+0000","last_fullsized":"2021-01-15T18:50:24.314258+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9033,"ondisk_log_size":9033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3756139,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9032,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9343","reported_seq":"9603","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:25.515919+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:25.515919+0000","last_peered":"2021-01-15T18:50:25.515919+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:25.515919+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9343,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3678208,"num_objects":449,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9348","reported_seq":"9954","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.332085+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:26.332085+0000","last_peered":"2021-01-15T18:50:26.332085+0000","last_clean":"2021-01-15T18:50:26.332085+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:26.332085+0000","last_undegraded":"2021-01-15T18:50:26.332085+0000","last_fullsized":"2021-01-15T18:50:26.332085+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9348,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3932326,"num_objects":481,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9075","reported_seq":"9300","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.294352+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:26.294352+0000","last_peered":"2021-01-15T18:50:26.294352+0000","last_clean":"2021-01-15T18:50:26.294352+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:26.294352+0000","last_undegraded":"2021-01-15T18:50:26.294352+0000","last_fullsized":"2021-01-15T18:50:26.294352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9075,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3694758,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9201","reported_seq":"12076","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:25.953579+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:25.953579+0000","last_peered":"2021-01-15T18:50:25.953579+0000","last_clean":"2021-01-15T18:50:25.953579+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:25.953579+0000","last_undegraded":"2021-01-15T18:50:25.953579+0000","last_fullsized":"2021-01-15T18:50:25.953579+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9201,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4292774,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9376","reported_seq":"12385","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.231659+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:26.231659+0000","last_peered":"2021-01-15T18:50:26.231659+0000","last_clean":"2021-01-15T18:50:26.231659+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:26.231659+0000","last_undegraded":"2021-01-15T18:50:26.231659+0000","last_fullsized":"2021-01-15T18:50:26.231659+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9376,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3936339,"num_objects":481,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9281","reported_seq":"12229","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.280498+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:26.280498+0000","last_peered":"2021-01-15T18:50:26.280498+0000","last_clean":"2021-01-15T18:50:26.280498+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:26.280498+0000","last_undegraded":"2021-01-15T18:50:26.280498+0000","last_fullsized":"2021-01-15T18:50:26.280498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9281,"ondisk_log_size":9281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3666003,"num_objects":448,"num_object_clones":0,"num_object_copies":1792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9292","reported_seq":"12247","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:25.933655+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:25.933655+0000","last_peered":"2021-01-15T18:50:25.933655+0000","last_clean":"2021-01-15T18:50:25.933655+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:25.933655+0000","last_undegraded":"2021-01-15T18:50:25.933655+0000","last_fullsized":"2021-01-15T18:50:25.933655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9292,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3813625,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9086","reported_seq":"11939","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.047152+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:26.047152+0000","last_peered":"2021-01-15T18:50:26.047152+0000","last_clean":"2021-01-15T18:50:26.047152+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:26.047152+0000","last_undegraded":"2021-01-15T18:50:26.047152+0000","last_fullsized":"2021-01-15T18:50:26.047152+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9086,"ondisk_log_size":9086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3834022,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9100","reported_seq":"11943","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.281154+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:26.281154+0000","last_peered":"2021-01-15T18:50:26.281154+0000","last_clean":"2021-01-15T18:50:26.281154+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:26.281154+0000","last_undegraded":"2021-01-15T18:50:26.281154+0000","last_fullsized":"2021-01-15T18:50:26.281154+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9100,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3825830,"num_objects":468,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9039","reported_seq":"11947","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:24.818676+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:24.818676+0000","last_peered":"2021-01-15T18:50:24.818676+0000","last_clean":"2021-01-15T18:50:24.818676+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:24.818676+0000","last_undegraded":"2021-01-15T18:50:24.818676+0000","last_fullsized":"2021-01-15T18:50:24.818676+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9039,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571878,"num_objects":437,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9416","reported_seq":"9656","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.048337+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:26.048337+0000","last_peered":"2021-01-15T18:50:26.048337+0000","last_clean":"2021-01-15T18:50:26.048337+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:26.048337+0000","last_undegraded":"2021-01-15T18:50:26.048337+0000","last_fullsized":"2021-01-15T18:50:26.048337+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9416,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3850406,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9585","reported_seq":"10629","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:26.189743+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:26.189743+0000","last_peered":"2021-01-15T18:50:26.189743+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:26.189743+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9585,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3744076,"num_objects":459,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9582,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9096","reported_seq":"11975","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:24.034796+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:24.034796+0000","last_peered":"2021-01-15T18:50:24.034796+0000","last_clean":"2021-01-15T18:50:24.034796+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:24.034796+0000","last_undegraded":"2021-01-15T18:50:24.034796+0000","last_fullsized":"2021-01-15T18:50:24.034796+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9096,"ondisk_log_size":9096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3629222,"num_objects":444,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9094,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":61163820,"num_objects":7481,"num_object_clones":0,"num_object_copies":29924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147244,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17817,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124002304,"data_stored":124002304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147271,"ondisk_log_size":147271,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306368,"kb_used_data":30284,"kb_used_omap":0,"kb_used_meta":276095,"kb_avail":94065472,"statfs":{"total":96636764160,"available":96323043328,"internally_reserved":0,"allocated":31010816,"data_stored":30727611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":282721332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,0,0,1,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":10.048},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":3.222}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.503}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.629},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.84499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307568,"kb_used_data":32092,"kb_used_omap":0,"kb_used_meta":275392,"kb_avail":94064272,"statfs":{"total":96636764160,"available":96321814528,"internally_reserved":0,"allocated":32862208,"data_stored":32579003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":282001408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":5.9160000000000004},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.379}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":5.056}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":8.8290000000000006},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":5.3369999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323608,"kb_used_data":31696,"kb_used_omap":0,"kb_used_meta":291903,"kb_avail":94048232,"statfs":{"total":96636764160,"available":96305389568,"internally_reserved":0,"allocated":32456704,"data_stored":32173499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298908724},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":2.427},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":2.528}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.45400000000000001},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.52600000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":0.63100000000000001}]}]},{"osd":1,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312892,"kb_used_data":30180,"kb_used_omap":0,"kb_used_meta":282687,"kb_avail":94058948,"statfs":{"total":96636764160,"available":96316362752,"internally_reserved":0,"allocated":30904320,"data_stored":30621115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":289471540},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":11.307},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.264}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.9430000000000001},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":1.3700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.464},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.5589999999999999}]}]}]}} 2021-01-15T18:50:28.439 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:30.342 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:30.343 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3590/29924 objects degraded (11.997%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7481,"data_bytes":61163820,"bytes_used":1280446464,"bytes_avail":385266610176,"bytes_total":386547056640,"degraded_objects":3590,"degraded_total":29924,"degraded_ratio":0.11997059216682261,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":149},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:30.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:31.208 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:31.211 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:31.229 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2021-01-15T18:50:30.677704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59795756,"num_objects":7314,"num_object_clones":0,"num_object_copies":29256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147411,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17817,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147438,"ondisk_log_size":147438,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1242112,"kb_used_data":116500,"kb_used_omap":2,"kb_used_meta":1125565,"kb_avail":376245248,"statfs":{"total":386547056640,"available":385275133952,"internally_reserved":0,"allocated":119296000,"data_stored":118163180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1152578716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,5,4,14,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":137,"apply_latency_ms":137,"commit_latency_ns":137000000,"apply_latency_ns":137000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16121856,"num_objects":-1968,"num_object_clones":0,"num_object_copies":-7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007455"},"pg_stats":[{"pgid":"1.f","version":"54'9013","reported_seq":"9285","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:25.994931+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:25.994931+0000","last_peered":"2021-01-15T18:50:25.994931+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:25.994931+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9013,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4055040,"num_objects":495,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-755}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'8987","reported_seq":"11925","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.332272+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:26.332272+0000","last_peered":"2021-01-15T18:50:26.332272+0000","last_clean":"2021-01-15T18:50:26.332272+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:26.332272+0000","last_undegraded":"2021-01-15T18:50:26.332272+0000","last_fullsized":"2021-01-15T18:50:26.332272+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":8987,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3883174,"num_objects":475,"num_object_clones":0,"num_object_copies":1900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9092","reported_seq":"12037","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:29.387641+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:29.387641+0000","last_peered":"2021-01-15T18:50:29.387641+0000","last_clean":"2021-01-15T18:50:29.387641+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:29.387641+0000","last_undegraded":"2021-01-15T18:50:29.387641+0000","last_fullsized":"2021-01-15T18:50:29.387641+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9092,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3272811,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9091,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9343","reported_seq":"9603","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:25.515919+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:25.515919+0000","last_peered":"2021-01-15T18:50:25.515919+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:25.515919+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9343,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3678208,"num_objects":449,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9348","reported_seq":"9954","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.332085+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:26.332085+0000","last_peered":"2021-01-15T18:50:26.332085+0000","last_clean":"2021-01-15T18:50:26.332085+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:26.332085+0000","last_undegraded":"2021-01-15T18:50:26.332085+0000","last_fullsized":"2021-01-15T18:50:26.332085+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9348,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3932326,"num_objects":481,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9075","reported_seq":"9300","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.294352+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:26.294352+0000","last_peered":"2021-01-15T18:50:26.294352+0000","last_clean":"2021-01-15T18:50:26.294352+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:26.294352+0000","last_undegraded":"2021-01-15T18:50:26.294352+0000","last_fullsized":"2021-01-15T18:50:26.294352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9075,"ondisk_log_size":9075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3694758,"num_objects":452,"num_object_clones":0,"num_object_copies":1808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9201","reported_seq":"12076","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:25.953579+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:25.953579+0000","last_peered":"2021-01-15T18:50:25.953579+0000","last_clean":"2021-01-15T18:50:25.953579+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:25.953579+0000","last_undegraded":"2021-01-15T18:50:25.953579+0000","last_fullsized":"2021-01-15T18:50:25.953579+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9201,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4292774,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9376","reported_seq":"12385","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.231659+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:26.231659+0000","last_peered":"2021-01-15T18:50:26.231659+0000","last_clean":"2021-01-15T18:50:26.231659+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:26.231659+0000","last_undegraded":"2021-01-15T18:50:26.231659+0000","last_fullsized":"2021-01-15T18:50:26.231659+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9376,"ondisk_log_size":9376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3936339,"num_objects":481,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9281","reported_seq":"12229","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.280498+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:26.280498+0000","last_peered":"2021-01-15T18:50:26.280498+0000","last_clean":"2021-01-15T18:50:26.280498+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:26.280498+0000","last_undegraded":"2021-01-15T18:50:26.280498+0000","last_fullsized":"2021-01-15T18:50:26.280498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9281,"ondisk_log_size":9281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3666003,"num_objects":448,"num_object_clones":0,"num_object_copies":1792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9292","reported_seq":"12247","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:25.933655+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:25.933655+0000","last_peered":"2021-01-15T18:50:25.933655+0000","last_clean":"2021-01-15T18:50:25.933655+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:25.933655+0000","last_undegraded":"2021-01-15T18:50:25.933655+0000","last_fullsized":"2021-01-15T18:50:25.933655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9292,"ondisk_log_size":9292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3813625,"num_objects":467,"num_object_clones":0,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9289,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9086","reported_seq":"11939","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.047152+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:26.047152+0000","last_peered":"2021-01-15T18:50:26.047152+0000","last_clean":"2021-01-15T18:50:26.047152+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:26.047152+0000","last_undegraded":"2021-01-15T18:50:26.047152+0000","last_fullsized":"2021-01-15T18:50:26.047152+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9086,"ondisk_log_size":9086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3834022,"num_objects":469,"num_object_clones":0,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9100","reported_seq":"11943","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.281154+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:26.281154+0000","last_peered":"2021-01-15T18:50:26.281154+0000","last_clean":"2021-01-15T18:50:26.281154+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:26.281154+0000","last_undegraded":"2021-01-15T18:50:26.281154+0000","last_fullsized":"2021-01-15T18:50:26.281154+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9100,"ondisk_log_size":9100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3825830,"num_objects":468,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9098","reported_seq":"12007","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:29.916698+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:29.916698+0000","last_peered":"2021-01-15T18:50:29.916698+0000","last_clean":"2021-01-15T18:50:29.916698+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:29.916698+0000","last_undegraded":"2021-01-15T18:50:29.916698+0000","last_fullsized":"2021-01-15T18:50:29.916698+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9098,"ondisk_log_size":9098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3088550,"num_objects":378,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9416","reported_seq":"9656","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:26.048337+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:26.048337+0000","last_peered":"2021-01-15T18:50:26.048337+0000","last_clean":"2021-01-15T18:50:26.048337+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:26.048337+0000","last_undegraded":"2021-01-15T18:50:26.048337+0000","last_fullsized":"2021-01-15T18:50:26.048337+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9416,"ondisk_log_size":9416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3850406,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9585","reported_seq":"10629","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:26.189743+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:26.189743+0000","last_peered":"2021-01-15T18:50:26.189743+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:26.189743+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9585,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3744076,"num_objects":459,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9582,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9145","reported_seq":"12023","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:29.387471+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:29.387471+0000","last_peered":"2021-01-15T18:50:29.387471+0000","last_clean":"2021-01-15T18:50:29.387471+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:29.387471+0000","last_undegraded":"2021-01-15T18:50:29.387471+0000","last_fullsized":"2021-01-15T18:50:29.387471+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9145,"ondisk_log_size":9145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3227814,"num_objects":395,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":59795756,"num_objects":7314,"num_object_clones":0,"num_object_copies":29256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147411,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17817,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116133888,"data_stored":116133888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147438,"ondisk_log_size":147438,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215235,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306368,"kb_used_data":30284,"kb_used_omap":0,"kb_used_meta":276095,"kb_avail":94065472,"statfs":{"total":96636764160,"available":96323043328,"internally_reserved":0,"allocated":31010816,"data_stored":30727611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":282721332},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,0,0,1,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":10.048},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":3.222}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.503}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.629},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.84499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313648,"kb_used_data":28260,"kb_used_omap":0,"kb_used_meta":285376,"kb_avail":94058192,"statfs":{"total":96636764160,"available":96315588608,"internally_reserved":0,"allocated":28938240,"data_stored":28655035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":292225024},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":10.028},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":4.6820000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":9.9390000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":31.202999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":9.7910000000000004},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":4.5730000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309204,"kb_used_data":27776,"kb_used_omap":0,"kb_used_meta":281407,"kb_avail":94062636,"statfs":{"total":96636764160,"available":96320139264,"internally_reserved":0,"allocated":28442624,"data_stored":28159419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":288160820},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":10.442},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":10.545}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":1.5549999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.878}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.6870000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.0269999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312892,"kb_used_data":30180,"kb_used_omap":0,"kb_used_meta":282687,"kb_avail":94058948,"statfs":{"total":96636764160,"available":96316362752,"internally_reserved":0,"allocated":30904320,"data_stored":30621115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":289471540},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,2,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":11.307},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.264}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":1.9430000000000001},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":1.3700000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.464},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.5589999999999999}]}]}]}} 2021-01-15T18:50:34.231 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:35.427 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:35.448 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:35.469 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2021-01-15T18:50:34.678675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53078316,"num_objects":6494,"num_object_clones":0,"num_object_copies":25976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148231,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17864,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148258,"ondisk_log_size":148258,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1240364,"kb_used_data":105680,"kb_used_omap":2,"kb_used_meta":1134589,"kb_avail":376246996,"statfs":{"total":386547056640,"available":385276923904,"internally_reserved":0,"allocated":108216320,"data_stored":107083500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1161819292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,6,21,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":138,"apply_latency_ms":138,"commit_latency_ns":138000000,"apply_latency_ns":138000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15777792,"num_objects":-1926,"num_object_clones":0,"num_object_copies":-7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1926,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002651"},"pg_stats":[{"pgid":"1.f","version":"54'9061","reported_seq":"9333","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:30.992214+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:30.992214+0000","last_peered":"2021-01-15T18:50:30.992214+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:30.992214+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9061,"ondisk_log_size":9061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3661824,"num_objects":447,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9039","reported_seq":"11977","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.807128+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:30.807128+0000","last_peered":"2021-01-15T18:50:30.807128+0000","last_clean":"2021-01-15T18:50:30.807128+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:30.807128+0000","last_undegraded":"2021-01-15T18:50:30.807128+0000","last_fullsized":"2021-01-15T18:50:30.807128+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9039,"ondisk_log_size":9039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3457190,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9133","reported_seq":"12078","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:34.180257+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:34.180257+0000","last_peered":"2021-01-15T18:50:34.180257+0000","last_clean":"2021-01-15T18:50:34.180257+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:34.180257+0000","last_undegraded":"2021-01-15T18:50:34.180257+0000","last_fullsized":"2021-01-15T18:50:34.180257+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9133,"ondisk_log_size":9133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2936939,"num_objects":360,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9132,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9400","reported_seq":"9660","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:30.766183+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:30.766183+0000","last_peered":"2021-01-15T18:50:30.766183+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:30.766183+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9400,"ondisk_log_size":9400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3211264,"num_objects":392,"num_object_clones":0,"num_object_copies":1568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-942}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9409","reported_seq":"10015","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.924253+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:30.924253+0000","last_peered":"2021-01-15T18:50:30.924253+0000","last_clean":"2021-01-15T18:50:30.924253+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:30.924253+0000","last_undegraded":"2021-01-15T18:50:30.924253+0000","last_fullsized":"2021-01-15T18:50:30.924253+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9409,"ondisk_log_size":9409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3432614,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9128","reported_seq":"9353","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.898280+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:30.898280+0000","last_peered":"2021-01-15T18:50:30.898280+0000","last_clean":"2021-01-15T18:50:30.898280+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:30.898280+0000","last_undegraded":"2021-01-15T18:50:30.898280+0000","last_fullsized":"2021-01-15T18:50:30.898280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9128,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3260582,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9255","reported_seq":"12130","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.844351+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:30.844351+0000","last_peered":"2021-01-15T18:50:30.844351+0000","last_clean":"2021-01-15T18:50:30.844351+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:30.844351+0000","last_undegraded":"2021-01-15T18:50:30.844351+0000","last_fullsized":"2021-01-15T18:50:30.844351+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9255,"ondisk_log_size":9255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3850406,"num_objects":471,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9425","reported_seq":"12436","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.894756+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:30.894756+0000","last_peered":"2021-01-15T18:50:30.894756+0000","last_clean":"2021-01-15T18:50:30.894756+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:30.894756+0000","last_undegraded":"2021-01-15T18:50:30.894756+0000","last_fullsized":"2021-01-15T18:50:30.894756+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9425,"ondisk_log_size":9425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3534931,"num_objects":432,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9343","reported_seq":"12292","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.835990+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:30.835990+0000","last_peered":"2021-01-15T18:50:30.835990+0000","last_clean":"2021-01-15T18:50:30.835990+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:30.835990+0000","last_undegraded":"2021-01-15T18:50:30.835990+0000","last_fullsized":"2021-01-15T18:50:30.835990+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9343,"ondisk_log_size":9343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3158099,"num_objects":386,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9355","reported_seq":"12310","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.932017+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:30.932017+0000","last_peered":"2021-01-15T18:50:30.932017+0000","last_clean":"2021-01-15T18:50:30.932017+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:30.932017+0000","last_undegraded":"2021-01-15T18:50:30.932017+0000","last_fullsized":"2021-01-15T18:50:30.932017+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9355,"ondisk_log_size":9355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3297529,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9134","reported_seq":"11987","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.879215+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:30.879215+0000","last_peered":"2021-01-15T18:50:30.879215+0000","last_clean":"2021-01-15T18:50:30.879215+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:30.879215+0000","last_undegraded":"2021-01-15T18:50:30.879215+0000","last_fullsized":"2021-01-15T18:50:30.879215+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9134,"ondisk_log_size":9134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3440806,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9132,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9156","reported_seq":"11999","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.943900+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:30.943900+0000","last_peered":"2021-01-15T18:50:30.943900+0000","last_clean":"2021-01-15T18:50:30.943900+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:30.943900+0000","last_undegraded":"2021-01-15T18:50:30.943900+0000","last_fullsized":"2021-01-15T18:50:30.943900+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9156,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3367078,"num_objects":412,"num_object_clones":0,"num_object_copies":1648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9098","reported_seq":"12007","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:29.916698+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:29.916698+0000","last_peered":"2021-01-15T18:50:29.916698+0000","last_clean":"2021-01-15T18:50:29.916698+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:29.916698+0000","last_undegraded":"2021-01-15T18:50:29.916698+0000","last_fullsized":"2021-01-15T18:50:29.916698+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9098,"ondisk_log_size":9098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3088550,"num_objects":378,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9468","reported_seq":"9708","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:30.939394+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:30.939394+0000","last_peered":"2021-01-15T18:50:30.939394+0000","last_clean":"2021-01-15T18:50:30.939394+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:30.939394+0000","last_undegraded":"2021-01-15T18:50:30.939394+0000","last_fullsized":"2021-01-15T18:50:30.939394+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9468,"ondisk_log_size":9468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3424422,"num_objects":419,"num_object_clones":0,"num_object_copies":1676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9466,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9652","reported_seq":"10791","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:31.196124+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:31.196124+0000","last_peered":"2021-01-15T18:50:31.196124+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:31.196124+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9652,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3195212,"num_objects":392,"num_object_clones":0,"num_object_copies":1568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-567}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9202","reported_seq":"12081","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:33.979560+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:33.979560+0000","last_peered":"2021-01-15T18:50:33.979560+0000","last_clean":"2021-01-15T18:50:33.979560+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:33.979560+0000","last_undegraded":"2021-01-15T18:50:33.979560+0000","last_fullsized":"2021-01-15T18:50:33.979560+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9202,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2760870,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53078316,"num_objects":6494,"num_object_clones":0,"num_object_copies":25976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148231,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17864,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":105345024,"data_stored":105345024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148258,"ondisk_log_size":148258,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312440,"kb_used_data":26472,"kb_used_omap":0,"kb_used_meta":285951,"kb_avail":94059400,"statfs":{"total":96636764160,"available":96316825600,"internally_reserved":0,"allocated":27107328,"data_stored":26824123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":292813876},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,8,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":6.2610000000000001},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":5.6459999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":8.6419999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":26.478999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":6.3710000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315960,"kb_used_data":24968,"kb_used_omap":0,"kb_used_meta":290944,"kb_avail":94055880,"statfs":{"total":96636764160,"available":96313221120,"internally_reserved":0,"allocated":25567232,"data_stored":25284027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":297926656},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.5199999999999996},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":6.6360000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":44.920999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":0.871},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.95599999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309204,"kb_used_data":27776,"kb_used_omap":0,"kb_used_meta":281407,"kb_avail":94062636,"statfs":{"total":96636764160,"available":96320139264,"internally_reserved":0,"allocated":28442624,"data_stored":28159419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":288160820},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":10.442},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":10.545}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":1.5549999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.878}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":6.6870000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.0269999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302760,"kb_used_data":26464,"kb_used_omap":0,"kb_used_meta":276287,"kb_avail":94069080,"statfs":{"total":96636764160,"available":96326737920,"internally_reserved":0,"allocated":27099136,"data_stored":26815931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":282917940},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,2,5,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":7.6340000000000003},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.6439999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":2.2400000000000002},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":1.7410000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.8360000000000001},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.931}]}]}]}} 2021-01-15T18:50:35.471 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:37.026 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:37.027 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3477/22884 objects degraded (15.194%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":687,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5721,"data_bytes":46745900,"bytes_used":1285574656,"bytes_avail":385261481984,"bytes_total":386547056640,"degraded_objects":3477,"degraded_total":22884,"degraded_ratio":0.15194022024121656,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":216},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2021-01-15T18:49:20.645350+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:37.123 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:37.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:38.554 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:38.595 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:38.615 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2021-01-15T18:50:36.679077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46745900,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149004,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17930,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149031,"ondisk_log_size":149031,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1255444,"kb_used_data":95576,"kb_used_omap":2,"kb_used_meta":1159677,"kb_avail":376231916,"statfs":{"total":386547056640,"available":385261481984,"internally_reserved":0,"allocated":97869824,"data_stored":96737004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1187509404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,11,8,18,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":137,"apply_latency_ms":137,"commit_latency_ns":137000000,"apply_latency_ns":137000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21291008,"num_objects":-2599,"num_object_clones":0,"num_object_copies":-10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"1.f","version":"54'9112","reported_seq":"9384","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:36.057995+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:36.057995+0000","last_peered":"2021-01-15T18:50:36.057995+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:36.057995+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9112,"ondisk_log_size":9112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3244032,"num_objects":396,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9112,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9083","reported_seq":"12021","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.200779+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:36.200779+0000","last_peered":"2021-01-15T18:50:36.200779+0000","last_clean":"2021-01-15T18:50:36.200779+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:36.200779+0000","last_undegraded":"2021-01-15T18:50:36.200779+0000","last_fullsized":"2021-01-15T18:50:36.200779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9083,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3096742,"num_objects":379,"num_object_clones":0,"num_object_copies":1516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9133","reported_seq":"12078","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:34.180257+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:34.180257+0000","last_peered":"2021-01-15T18:50:34.180257+0000","last_clean":"2021-01-15T18:50:34.180257+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:34.180257+0000","last_undegraded":"2021-01-15T18:50:34.180257+0000","last_fullsized":"2021-01-15T18:50:34.180257+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9133,"ondisk_log_size":9133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2936939,"num_objects":360,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9132,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9453","reported_seq":"9713","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:36.191876+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:36.191876+0000","last_peered":"2021-01-15T18:50:36.191876+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:36.191876+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9453,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2777088,"num_objects":339,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9463","reported_seq":"10069","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.193499+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:36.193499+0000","last_peered":"2021-01-15T18:50:36.193499+0000","last_clean":"2021-01-15T18:50:36.193499+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:36.193499+0000","last_undegraded":"2021-01-15T18:50:36.193499+0000","last_fullsized":"2021-01-15T18:50:36.193499+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9463,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2990246,"num_objects":366,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9195","reported_seq":"9418","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.315255+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:36.315255+0000","last_peered":"2021-01-15T18:50:36.315255+0000","last_clean":"2021-01-15T18:50:36.315255+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:36.315255+0000","last_undegraded":"2021-01-15T18:50:36.315255+0000","last_fullsized":"2021-01-15T18:50:36.315255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9195,"ondisk_log_size":9195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2711718,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9308","reported_seq":"12183","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.136489+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:36.136489+0000","last_peered":"2021-01-15T18:50:36.136489+0000","last_clean":"2021-01-15T18:50:36.136489+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:36.136489+0000","last_undegraded":"2021-01-15T18:50:36.136489+0000","last_fullsized":"2021-01-15T18:50:36.136489+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9308,"ondisk_log_size":9308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3416230,"num_objects":418,"num_object_clones":0,"num_object_copies":1672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9306,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9488","reported_seq":"12497","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.342318+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:36.342318+0000","last_peered":"2021-01-15T18:50:36.342318+0000","last_clean":"2021-01-15T18:50:36.342318+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:36.342318+0000","last_undegraded":"2021-01-15T18:50:36.342318+0000","last_fullsized":"2021-01-15T18:50:36.342318+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9488,"ondisk_log_size":9488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3018835,"num_objects":369,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9403","reported_seq":"12352","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.325938+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:36.325938+0000","last_peered":"2021-01-15T18:50:36.325938+0000","last_clean":"2021-01-15T18:50:36.325938+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:36.325938+0000","last_undegraded":"2021-01-15T18:50:36.325938+0000","last_fullsized":"2021-01-15T18:50:36.325938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9403,"ondisk_log_size":9403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2666579,"num_objects":326,"num_object_clones":0,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9403","reported_seq":"12357","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:35.904464+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:35.904464+0000","last_peered":"2021-01-15T18:50:35.904464+0000","last_clean":"2021-01-15T18:50:35.904464+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:35.904464+0000","last_undegraded":"2021-01-15T18:50:35.904464+0000","last_fullsized":"2021-01-15T18:50:35.904464+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9403,"ondisk_log_size":9403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2904313,"num_objects":356,"num_object_clones":0,"num_object_copies":1424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9196","reported_seq":"12049","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.118271+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:36.118271+0000","last_peered":"2021-01-15T18:50:36.118271+0000","last_clean":"2021-01-15T18:50:36.118271+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:36.118271+0000","last_undegraded":"2021-01-15T18:50:36.118271+0000","last_fullsized":"2021-01-15T18:50:36.118271+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9196,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2932902,"num_objects":359,"num_object_clones":0,"num_object_copies":1436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9225","reported_seq":"12068","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.152674+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:36.152674+0000","last_peered":"2021-01-15T18:50:36.152674+0000","last_clean":"2021-01-15T18:50:36.152674+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:36.152674+0000","last_undegraded":"2021-01-15T18:50:36.152674+0000","last_fullsized":"2021-01-15T18:50:36.152674+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9225,"ondisk_log_size":9225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2801830,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9223,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9154","reported_seq":"12063","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:34.935480+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:34.935480+0000","last_peered":"2021-01-15T18:50:34.935480+0000","last_clean":"2021-01-15T18:50:34.935480+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:34.935480+0000","last_undegraded":"2021-01-15T18:50:34.935480+0000","last_fullsized":"2021-01-15T18:50:34.935480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9154,"ondisk_log_size":9154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2629798,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9517","reported_seq":"9757","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:36.156725+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:36.156725+0000","last_peered":"2021-01-15T18:50:36.156725+0000","last_clean":"2021-01-15T18:50:36.156725+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:36.156725+0000","last_undegraded":"2021-01-15T18:50:36.156725+0000","last_fullsized":"2021-01-15T18:50:36.156725+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9517,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3023014,"num_objects":370,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9696","reported_seq":"10967","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:36.118462+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:36.118462+0000","last_peered":"2021-01-15T18:50:36.118462+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:36.118462+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9696,"ondisk_log_size":9696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2834764,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9202","reported_seq":"12081","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:33.979560+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:33.979560+0000","last_peered":"2021-01-15T18:50:33.979560+0000","last_clean":"2021-01-15T18:50:33.979560+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:33.979560+0000","last_undegraded":"2021-01-15T18:50:33.979560+0000","last_fullsized":"2021-01-15T18:50:33.979560+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9202,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2760870,"num_objects":338,"num_object_clones":0,"num_object_copies":1352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":46745900,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149004,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17930,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149031,"ondisk_log_size":149031,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215237,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314760,"kb_used_data":23116,"kb_used_omap":0,"kb_used_meta":291583,"kb_avail":94057080,"statfs":{"total":96636764160,"available":96314449920,"internally_reserved":0,"allocated":23670784,"data_stored":23387579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298581044},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.72399999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":3.702}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":3.9239999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.8199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315960,"kb_used_data":24968,"kb_used_omap":0,"kb_used_meta":290944,"kb_avail":94055880,"statfs":{"total":96636764160,"available":96313221120,"internally_reserved":0,"allocated":25567232,"data_stored":25284027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":297926656},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":6.5199999999999996},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:49:38 2021","interfaces":[{"interface":"back","average":{"1min":3.9460000000000002,"5min":3.4620000000000002,"15min":2.847},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.565,"5min":24.071000000000002,"15min":24.071000000000002},"last":6.6360000000000001},{"interface":"front","average":{"1min":2.879,"5min":2.2919999999999998,"15min":2.1859999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":17.071000000000002,"5min":17.071000000000002,"15min":20.98},"last":44.920999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":0.871},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.95599999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315520,"kb_used_data":24408,"kb_used_omap":0,"kb_used_meta":291071,"kb_avail":94056320,"statfs":{"total":96636764160,"available":96313671680,"internally_reserved":0,"allocated":24993792,"data_stored":24710587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298056756},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":37.043999999999997},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":37.140000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.621}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.22},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":4.0620000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309204,"kb_used_data":23084,"kb_used_omap":0,"kb_used_meta":286079,"kb_avail":94062636,"statfs":{"total":96636764160,"available":96320139264,"internally_reserved":0,"allocated":23638016,"data_stored":23354811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":292944948},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,4,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":5.7220000000000004},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":30.181999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":5.5339999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.97599999999999998}]}]}]}} 2021-01-15T18:50:41.619 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:43.017 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:43.018 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:43.030 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2021-01-15T18:50:42.683807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39192876,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3426,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149926,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17981,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149953,"ondisk_log_size":149953,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1210848,"kb_used_data":81348,"kb_used_omap":2,"kb_used_meta":1129277,"kb_avail":376276512,"statfs":{"total":386547056640,"available":385307148288,"internally_reserved":0,"allocated":83300352,"data_stored":82167532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1156379804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,11,16,2,15,21,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":140,"apply_latency_ms":140,"commit_latency_ns":140000000,"apply_latency_ns":140000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20602880,"num_objects":-2515,"num_object_clones":0,"num_object_copies":-10060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006102"},"pg_stats":[{"pgid":"1.f","version":"54'9186","reported_seq":"9458","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:41.038376+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:41.038376+0000","last_peered":"2021-01-15T18:50:41.038376+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:41.038376+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9186,"ondisk_log_size":9186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2637824,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9186,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9153","reported_seq":"12091","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.158539+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:41.158539+0000","last_peered":"2021-01-15T18:50:41.158539+0000","last_clean":"2021-01-15T18:50:41.158539+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:41.158539+0000","last_undegraded":"2021-01-15T18:50:41.158539+0000","last_fullsized":"2021-01-15T18:50:41.158539+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9153,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2523302,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9203","reported_seq":"12148","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:39.393517+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:39.393517+0000","last_peered":"2021-01-15T18:50:39.393517+0000","last_clean":"2021-01-15T18:50:39.393517+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:39.393517+0000","last_undegraded":"2021-01-15T18:50:39.393517+0000","last_fullsized":"2021-01-15T18:50:39.393517+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9203,"ondisk_log_size":9203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2363499,"num_objects":290,"num_object_clones":0,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9202,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9497","reported_seq":"9757","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:40.792310+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:40.792310+0000","last_peered":"2021-01-15T18:50:40.792310+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:40.792310+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9497,"ondisk_log_size":9497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2416640,"num_objects":295,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9518","reported_seq":"10124","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.292911+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:41.292911+0000","last_peered":"2021-01-15T18:50:41.292911+0000","last_clean":"2021-01-15T18:50:41.292911+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:41.292911+0000","last_undegraded":"2021-01-15T18:50:41.292911+0000","last_fullsized":"2021-01-15T18:50:41.292911+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9518,"ondisk_log_size":9518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2539686,"num_objects":311,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9516,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9252","reported_seq":"9477","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.379890+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:41.379890+0000","last_peered":"2021-01-15T18:50:41.379890+0000","last_clean":"2021-01-15T18:50:41.379890+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:41.379890+0000","last_undegraded":"2021-01-15T18:50:41.379890+0000","last_fullsized":"2021-01-15T18:50:41.379890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9252,"ondisk_log_size":9252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244774,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9371","reported_seq":"12246","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.170231+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:41.170231+0000","last_peered":"2021-01-15T18:50:41.170231+0000","last_clean":"2021-01-15T18:50:41.170231+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:41.170231+0000","last_undegraded":"2021-01-15T18:50:41.170231+0000","last_fullsized":"2021-01-15T18:50:41.170231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9371,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2900134,"num_objects":355,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9542","reported_seq":"12553","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.028105+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:41.028105+0000","last_peered":"2021-01-15T18:50:41.028105+0000","last_clean":"2021-01-15T18:50:41.028105+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:41.028105+0000","last_undegraded":"2021-01-15T18:50:41.028105+0000","last_fullsized":"2021-01-15T18:50:41.028105+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9542,"ondisk_log_size":9542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2576467,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9447","reported_seq":"12396","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.295105+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:41.295105+0000","last_peered":"2021-01-15T18:50:41.295105+0000","last_clean":"2021-01-15T18:50:41.295105+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:41.295105+0000","last_undegraded":"2021-01-15T18:50:41.295105+0000","last_fullsized":"2021-01-15T18:50:41.295105+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9447,"ondisk_log_size":9447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2306131,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9446,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9460","reported_seq":"12415","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.203979+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:41.203979+0000","last_peered":"2021-01-15T18:50:41.203979+0000","last_clean":"2021-01-15T18:50:41.203979+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:41.203979+0000","last_undegraded":"2021-01-15T18:50:41.203979+0000","last_fullsized":"2021-01-15T18:50:41.203979+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437369,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9258","reported_seq":"12111","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.152117+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:41.152117+0000","last_peered":"2021-01-15T18:50:41.152117+0000","last_clean":"2021-01-15T18:50:41.152117+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:41.152117+0000","last_undegraded":"2021-01-15T18:50:41.152117+0000","last_fullsized":"2021-01-15T18:50:41.152117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9258,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2424998,"num_objects":297,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9289","reported_seq":"12132","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.295302+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:41.295302+0000","last_peered":"2021-01-15T18:50:41.295302+0000","last_clean":"2021-01-15T18:50:41.295302+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:41.295302+0000","last_undegraded":"2021-01-15T18:50:41.295302+0000","last_fullsized":"2021-01-15T18:50:41.295302+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9289,"ondisk_log_size":9289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2277542,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9207","reported_seq":"12116","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:39.729773+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:39.729773+0000","last_peered":"2021-01-15T18:50:39.729773+0000","last_clean":"2021-01-15T18:50:39.729773+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:39.729773+0000","last_undegraded":"2021-01-15T18:50:39.729773+0000","last_fullsized":"2021-01-15T18:50:39.729773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9207,"ondisk_log_size":9207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2195622,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9570","reported_seq":"9810","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.243390+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:41.243390+0000","last_peered":"2021-01-15T18:50:41.243390+0000","last_clean":"2021-01-15T18:50:41.243390+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:41.243390+0000","last_undegraded":"2021-01-15T18:50:41.243390+0000","last_fullsized":"2021-01-15T18:50:41.243390+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9570,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2588838,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9759","reported_seq":"11132","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:41.239824+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:41.239824+0000","last_peered":"2021-01-15T18:50:41.239824+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:41.239824+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9759,"ondisk_log_size":9759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2318668,"num_objects":285,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9241","reported_seq":"12120","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:39.228491+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:39.228491+0000","last_peered":"2021-01-15T18:50:39.228491+0000","last_clean":"2021-01-15T18:50:39.228491+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:39.228491+0000","last_undegraded":"2021-01-15T18:50:39.228491+0000","last_fullsized":"2021-01-15T18:50:39.228491+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9241,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2441382,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":39192876,"num_objects":4799,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3426,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149926,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17981,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":79781888,"data_stored":79781888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149953,"ondisk_log_size":149953,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304900,"kb_used_data":20116,"kb_used_omap":0,"kb_used_meta":284735,"kb_avail":94066940,"statfs":{"total":96636764160,"available":96324546560,"internally_reserved":0,"allocated":20598784,"data_stored":20315579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291568692},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,7,13,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":2.7480000000000002},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.435}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":2.548},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":2.899}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4580000000000002},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":2.6509999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305708,"kb_used_data":21244,"kb_used_omap":0,"kb_used_meta":284416,"kb_avail":94066132,"statfs":{"total":96636764160,"available":96323719168,"internally_reserved":0,"allocated":21753856,"data_stored":21470651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":291241984},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,7,0,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.361},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":13.435}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":32.691000000000003},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":0.91300000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305292,"kb_used_data":20604,"kb_used_omap":0,"kb_used_meta":284607,"kb_avail":94066548,"statfs":{"total":96636764160,"available":96324145152,"internally_reserved":0,"allocated":21098496,"data_stored":20815291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291437620},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.65300000000000002},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.746}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":4.8330000000000002},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":7.9630000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.0209999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":9.2799999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294948,"kb_used_data":19384,"kb_used_omap":0,"kb_used_meta":275519,"kb_avail":94076892,"statfs":{"total":96636764160,"available":96334737408,"internally_reserved":0,"allocated":19849216,"data_stored":19566011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":282131508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.997},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":12.177}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":2.5880000000000001},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":12.942}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":12.285},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":12.371}]}]}]}} 2021-01-15T18:50:43.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:44.485 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:44.486 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3426/19196 objects degraded (17.847%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":694,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4799,"data_bytes":39192876,"bytes_used":1239908352,"bytes_avail":385307148288,"bytes_total":386547056640,"degraded_objects":3426,"degraded_total":19196,"degraded_ratio":0.17847468222546364,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":209},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:44.499 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:44.499 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:45.479 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:45.510 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2021-01-15T18:50:44.685808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38422828,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3426,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150020,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17981,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150047,"ondisk_log_size":150047,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1212904,"kb_used_data":77668,"kb_used_omap":2,"kb_used_meta":1135037,"kb_avail":376274456,"statfs":{"total":386547056640,"available":385305042944,"internally_reserved":0,"allocated":79532032,"data_stored":78399212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1162278044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,10,6,15,15,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":134,"apply_latency_ms":134,"commit_latency_ns":134000000,"apply_latency_ns":134000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15458304,"num_objects":-1887,"num_object_clones":0,"num_object_copies":-7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007564"},"pg_stats":[{"pgid":"1.f","version":"54'9186","reported_seq":"9458","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:41.038376+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:41.038376+0000","last_peered":"2021-01-15T18:50:41.038376+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:41.038376+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9186,"ondisk_log_size":9186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2637824,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9186,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9153","reported_seq":"12091","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.158539+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:41.158539+0000","last_peered":"2021-01-15T18:50:41.158539+0000","last_clean":"2021-01-15T18:50:41.158539+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:41.158539+0000","last_undegraded":"2021-01-15T18:50:41.158539+0000","last_fullsized":"2021-01-15T18:50:41.158539+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9153,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2523302,"num_objects":309,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9237","reported_seq":"12182","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:44.283884+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:44.283884+0000","last_peered":"2021-01-15T18:50:44.283884+0000","last_clean":"2021-01-15T18:50:44.283884+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:44.283884+0000","last_undegraded":"2021-01-15T18:50:44.283884+0000","last_fullsized":"2021-01-15T18:50:44.283884+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9237,"ondisk_log_size":9237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2084971,"num_objects":256,"num_object_clones":0,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9236,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9497","reported_seq":"9757","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:40.792310+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:40.792310+0000","last_peered":"2021-01-15T18:50:40.792310+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:40.792310+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9497,"ondisk_log_size":9497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2416640,"num_objects":295,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1039}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9518","reported_seq":"10124","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.292911+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:41.292911+0000","last_peered":"2021-01-15T18:50:41.292911+0000","last_clean":"2021-01-15T18:50:41.292911+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:41.292911+0000","last_undegraded":"2021-01-15T18:50:41.292911+0000","last_fullsized":"2021-01-15T18:50:41.292911+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9518,"ondisk_log_size":9518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2539686,"num_objects":311,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9516,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9252","reported_seq":"9477","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.379890+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:41.379890+0000","last_peered":"2021-01-15T18:50:41.379890+0000","last_clean":"2021-01-15T18:50:41.379890+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:41.379890+0000","last_undegraded":"2021-01-15T18:50:41.379890+0000","last_fullsized":"2021-01-15T18:50:41.379890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9252,"ondisk_log_size":9252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244774,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9371","reported_seq":"12246","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.170231+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:41.170231+0000","last_peered":"2021-01-15T18:50:41.170231+0000","last_clean":"2021-01-15T18:50:41.170231+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:41.170231+0000","last_undegraded":"2021-01-15T18:50:41.170231+0000","last_fullsized":"2021-01-15T18:50:41.170231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9371,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2900134,"num_objects":355,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9542","reported_seq":"12553","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.028105+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:41.028105+0000","last_peered":"2021-01-15T18:50:41.028105+0000","last_clean":"2021-01-15T18:50:41.028105+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:41.028105+0000","last_undegraded":"2021-01-15T18:50:41.028105+0000","last_fullsized":"2021-01-15T18:50:41.028105+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9542,"ondisk_log_size":9542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2576467,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9447","reported_seq":"12396","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.295105+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:41.295105+0000","last_peered":"2021-01-15T18:50:41.295105+0000","last_clean":"2021-01-15T18:50:41.295105+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:41.295105+0000","last_undegraded":"2021-01-15T18:50:41.295105+0000","last_fullsized":"2021-01-15T18:50:41.295105+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9447,"ondisk_log_size":9447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2306131,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9446,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9460","reported_seq":"12415","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.203979+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:41.203979+0000","last_peered":"2021-01-15T18:50:41.203979+0000","last_clean":"2021-01-15T18:50:41.203979+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:41.203979+0000","last_undegraded":"2021-01-15T18:50:41.203979+0000","last_fullsized":"2021-01-15T18:50:41.203979+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437369,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9258","reported_seq":"12111","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.152117+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:41.152117+0000","last_peered":"2021-01-15T18:50:41.152117+0000","last_clean":"2021-01-15T18:50:41.152117+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:41.152117+0000","last_undegraded":"2021-01-15T18:50:41.152117+0000","last_fullsized":"2021-01-15T18:50:41.152117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9258,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2424998,"num_objects":297,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9289","reported_seq":"12132","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.295302+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:41.295302+0000","last_peered":"2021-01-15T18:50:41.295302+0000","last_clean":"2021-01-15T18:50:41.295302+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:41.295302+0000","last_undegraded":"2021-01-15T18:50:41.295302+0000","last_fullsized":"2021-01-15T18:50:41.295302+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9289,"ondisk_log_size":9289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2277542,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9207","reported_seq":"12116","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:39.729773+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:39.729773+0000","last_peered":"2021-01-15T18:50:39.729773+0000","last_clean":"2021-01-15T18:50:39.729773+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:39.729773+0000","last_undegraded":"2021-01-15T18:50:39.729773+0000","last_fullsized":"2021-01-15T18:50:39.729773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9207,"ondisk_log_size":9207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2195622,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9570","reported_seq":"9810","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:41.243390+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:41.243390+0000","last_peered":"2021-01-15T18:50:41.243390+0000","last_clean":"2021-01-15T18:50:41.243390+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:41.243390+0000","last_undegraded":"2021-01-15T18:50:41.243390+0000","last_fullsized":"2021-01-15T18:50:41.243390+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9570,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2588838,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9568,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9759","reported_seq":"11132","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:41.239824+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:41.239824+0000","last_peered":"2021-01-15T18:50:41.239824+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:41.239824+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9759,"ondisk_log_size":9759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2318668,"num_objects":285,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-557}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9301","reported_seq":"12180","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:44.324198+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:44.324198+0000","last_peered":"2021-01-15T18:50:44.324198+0000","last_clean":"2021-01-15T18:50:44.324198+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:44.324198+0000","last_undegraded":"2021-01-15T18:50:44.324198+0000","last_fullsized":"2021-01-15T18:50:44.324198+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9301,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1949862,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":38422828,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3426,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150020,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17981,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":76111872,"data_stored":76111872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150047,"ondisk_log_size":150047,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215238,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304900,"kb_used_data":20116,"kb_used_omap":0,"kb_used_meta":284735,"kb_avail":94066940,"statfs":{"total":96636764160,"available":96324546560,"internally_reserved":0,"allocated":20598784,"data_stored":20315579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291568692},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,7,13,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":2.7480000000000002},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.435}]},{"osd":1,"last update":"Fri Jan 15 18:49:44 2021","interfaces":[{"interface":"back","average":{"1min":8.2739999999999991,"5min":4.8259999999999996,"15min":3.9980000000000002},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":25.114999999999998,"5min":27.218,"15min":30.129000000000001},"last":2.548},{"interface":"front","average":{"1min":8.9619999999999997,"5min":5.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.68400000000000005,"5min":0.57899999999999996,"15min":0.45500000000000002},"max":{"1min":38.700000000000003,"5min":38.700000000000003,"15min":38.700000000000003},"last":2.899}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4580000000000002},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":2.6509999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307764,"kb_used_data":17564,"kb_used_omap":0,"kb_used_meta":290176,"kb_avail":94064076,"statfs":{"total":96636764160,"available":96321613824,"internally_reserved":0,"allocated":17985536,"data_stored":17702331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":297140224},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":6.476},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":5.9619999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":10.826000000000001},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":23.440000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305292,"kb_used_data":20604,"kb_used_omap":0,"kb_used_meta":284607,"kb_avail":94066548,"statfs":{"total":96636764160,"available":96324145152,"internally_reserved":0,"allocated":21098496,"data_stored":20815291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291437620},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:49:39 2021","interfaces":[{"interface":"back","average":{"1min":5.5940000000000003,"5min":6.6020000000000003,"15min":4.0499999999999998},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":23.111000000000001,"5min":50.948,"15min":50.948},"last":0.65300000000000002},{"interface":"front","average":{"1min":5.9119999999999999,"5min":5.4660000000000002,"15min":3.7669999999999999},"min":{"1min":0.56999999999999995,"5min":0.375,"15min":0.375},"max":{"1min":23.204000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.746}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":4.8330000000000002},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":7.9630000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":4.0209999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":9.2799999999999994}]}]},{"osd":1,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294948,"kb_used_data":19384,"kb_used_omap":0,"kb_used_meta":275519,"kb_avail":94076892,"statfs":{"total":96636764160,"available":96334737408,"internally_reserved":0,"allocated":19849216,"data_stored":19566011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":282131508},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.997},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":12.177}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":2.5880000000000001},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":12.942}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":12.285},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":12.371}]}]}]}} 2021-01-15T18:50:48.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:49.847 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:49.916 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:49.942 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2021-01-15T18:50:48.686636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32049452,"num_objects":3927,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150798,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18028,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150825,"ondisk_log_size":150825,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1227956,"kb_used_data":67816,"kb_used_omap":2,"kb_used_meta":1160061,"kb_avail":376259404,"statfs":{"total":386547056640,"available":385289629696,"internally_reserved":0,"allocated":69443584,"data_stored":68310764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1187902620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,8,11,4,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":132,"apply_latency_ms":132,"commit_latency_ns":132000000,"apply_latency_ns":132000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14696448,"num_objects":-1794,"num_object_clones":0,"num_object_copies":-7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1794,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007559"},"pg_stats":[{"pgid":"1.f","version":"54'9254","reported_seq":"9526","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:46.247789+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:46.247789+0000","last_peered":"2021-01-15T18:50:46.247789+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:46.247789+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9254,"ondisk_log_size":9254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2080768,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-996}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9216","reported_seq":"12154","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.067701+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:46.067701+0000","last_peered":"2021-01-15T18:50:46.067701+0000","last_clean":"2021-01-15T18:50:46.067701+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:46.067701+0000","last_undegraded":"2021-01-15T18:50:46.067701+0000","last_fullsized":"2021-01-15T18:50:46.067701+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9216,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2007206,"num_objects":246,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9237","reported_seq":"12182","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:44.283884+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:44.283884+0000","last_peered":"2021-01-15T18:50:44.283884+0000","last_clean":"2021-01-15T18:50:44.283884+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:44.283884+0000","last_undegraded":"2021-01-15T18:50:44.283884+0000","last_fullsized":"2021-01-15T18:50:44.283884+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9237,"ondisk_log_size":9237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2084971,"num_objects":256,"num_object_clones":0,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9236,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9549","reported_seq":"9809","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:46.205949+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:46.205949+0000","last_peered":"2021-01-15T18:50:46.205949+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:46.205949+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9549,"ondisk_log_size":9549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1990656,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9589","reported_seq":"10195","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.207881+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:46.207881+0000","last_peered":"2021-01-15T18:50:46.207881+0000","last_clean":"2021-01-15T18:50:46.207881+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:46.207881+0000","last_undegraded":"2021-01-15T18:50:46.207881+0000","last_fullsized":"2021-01-15T18:50:46.207881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9589,"ondisk_log_size":9589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1958054,"num_objects":240,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9587,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9299","reported_seq":"9524","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.163337+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:46.163337+0000","last_peered":"2021-01-15T18:50:46.163337+0000","last_clean":"2021-01-15T18:50:46.163337+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:46.163337+0000","last_undegraded":"2021-01-15T18:50:46.163337+0000","last_fullsized":"2021-01-15T18:50:46.163337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9299,"ondisk_log_size":9299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1859750,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9297,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9436","reported_seq":"12311","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.099957+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:46.099957+0000","last_peered":"2021-01-15T18:50:46.099957+0000","last_clean":"2021-01-15T18:50:46.099957+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:46.099957+0000","last_undegraded":"2021-01-15T18:50:46.099957+0000","last_fullsized":"2021-01-15T18:50:46.099957+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9436,"ondisk_log_size":9436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2367654,"num_objects":290,"num_object_clones":0,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9599","reported_seq":"12610","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.179954+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:46.179954+0000","last_peered":"2021-01-15T18:50:46.179954+0000","last_clean":"2021-01-15T18:50:46.179954+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:46.179954+0000","last_undegraded":"2021-01-15T18:50:46.179954+0000","last_fullsized":"2021-01-15T18:50:46.179954+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9599,"ondisk_log_size":9599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2109523,"num_objects":258,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9598,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9495","reported_seq":"12444","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.057530+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:46.057530+0000","last_peered":"2021-01-15T18:50:46.057530+0000","last_clean":"2021-01-15T18:50:46.057530+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:46.057530+0000","last_undegraded":"2021-01-15T18:50:46.057530+0000","last_fullsized":"2021-01-15T18:50:46.057530+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9495,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1912915,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9510","reported_seq":"12464","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.188903+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:46.188903+0000","last_peered":"2021-01-15T18:50:46.188903+0000","last_clean":"2021-01-15T18:50:46.188903+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:46.188903+0000","last_undegraded":"2021-01-15T18:50:46.188903+0000","last_fullsized":"2021-01-15T18:50:46.188903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9510,"ondisk_log_size":9510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2027769,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9507,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9312","reported_seq":"12165","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.220924+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:46.220924+0000","last_peered":"2021-01-15T18:50:46.220924+0000","last_clean":"2021-01-15T18:50:46.220924+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:46.220924+0000","last_undegraded":"2021-01-15T18:50:46.220924+0000","last_fullsized":"2021-01-15T18:50:46.220924+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9312,"ondisk_log_size":9312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1982630,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9329","reported_seq":"12172","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.193262+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:46.193262+0000","last_peered":"2021-01-15T18:50:46.193262+0000","last_clean":"2021-01-15T18:50:46.193262+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:46.193262+0000","last_undegraded":"2021-01-15T18:50:46.193262+0000","last_fullsized":"2021-01-15T18:50:46.193262+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9329,"ondisk_log_size":9329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1949862,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9258","reported_seq":"12167","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:44.885569+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:44.885569+0000","last_peered":"2021-01-15T18:50:44.885569+0000","last_clean":"2021-01-15T18:50:44.885569+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:44.885569+0000","last_undegraded":"2021-01-15T18:50:44.885569+0000","last_fullsized":"2021-01-15T18:50:44.885569+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9258,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1777830,"num_objects":218,"num_object_clones":0,"num_object_copies":872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9627","reported_seq":"9867","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:46.208049+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:46.208049+0000","last_peered":"2021-01-15T18:50:46.208049+0000","last_clean":"2021-01-15T18:50:46.208049+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:46.208049+0000","last_undegraded":"2021-01-15T18:50:46.208049+0000","last_fullsized":"2021-01-15T18:50:46.208049+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9627,"ondisk_log_size":9627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2121894,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9625,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9814","reported_seq":"11280","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:46.233406+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:46.233406+0000","last_peered":"2021-01-15T18:50:46.233406+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:46.233406+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9814,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1868108,"num_objects":230,"num_object_clones":0,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":697,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9301","reported_seq":"12180","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:44.324198+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:44.324198+0000","last_peered":"2021-01-15T18:50:44.324198+0000","last_clean":"2021-01-15T18:50:44.324198+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:44.324198+0000","last_undegraded":"2021-01-15T18:50:44.324198+0000","last_fullsized":"2021-01-15T18:50:44.324198+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9301,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1949862,"num_objects":239,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":32049452,"num_objects":3927,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150798,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18028,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65347584,"data_stored":65347584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150825,"ondisk_log_size":150825,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307040,"kb_used_data":16584,"kb_used_omap":0,"kb_used_meta":290431,"kb_avail":94064800,"statfs":{"total":96636764160,"available":96322355200,"internally_reserved":0,"allocated":16982016,"data_stored":16698811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":297401396},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":3.1200000000000001},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.62}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":1.21}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":6.0629999999999997},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.9380000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307764,"kb_used_data":17564,"kb_used_omap":0,"kb_used_meta":290176,"kb_avail":94064076,"statfs":{"total":96636764160,"available":96321613824,"internally_reserved":0,"allocated":17985536,"data_stored":17702331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":297140224},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":6.476},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":5.9619999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":10.826000000000001},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":23.440000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311500,"kb_used_data":17092,"kb_used_omap":0,"kb_used_meta":294399,"kb_avail":94060340,"statfs":{"total":96636764160,"available":96317788160,"internally_reserved":0,"allocated":17502208,"data_stored":17219003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":301464628},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":16.111000000000001},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":16.213999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":1.9650000000000001},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":11.319000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":11.218999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301652,"kb_used_data":16576,"kb_used_omap":0,"kb_used_meta":285055,"kb_avail":94070188,"statfs":{"total":96636764160,"available":96327872512,"internally_reserved":0,"allocated":16973824,"data_stored":16690619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291896372},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,0,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":1.863}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":1.5469999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.7629999999999999}]}]}]}} 2021-01-15T18:50:49.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:51.443 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:51.443 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3379/15096 objects degraded (22.383%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":701,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3774,"data_bytes":30796076,"bytes_used":1232498688,"bytes_avail":385314557952,"bytes_total":386547056640,"degraded_objects":3379,"degraded_total":15096,"degraded_ratio":0.22383412824589294,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:51.532 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:51.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:52.896 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:52.955 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:53.000 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2021-01-15T18:50:52.690835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24488236,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18066,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151748,"ondisk_log_size":151748,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1216112,"kb_used_data":53192,"kb_used_omap":2,"kb_used_meta":1162813,"kb_avail":376271248,"statfs":{"total":386547056640,"available":385301757952,"internally_reserved":0,"allocated":54468608,"data_stored":53335788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1190720668},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,4,10,23,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20930560,"num_objects":-2555,"num_object_clones":0,"num_object_copies":-10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010935"},"pg_stats":[{"pgid":"1.f","version":"54'9307","reported_seq":"9578","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:51.055290+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:51.055290+0000","last_peered":"2021-01-15T18:50:51.055290+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:51.055290+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9307,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1646592,"num_objects":201,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9279","reported_seq":"12217","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.274598+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:51.274598+0000","last_peered":"2021-01-15T18:50:51.274598+0000","last_clean":"2021-01-15T18:50:51.274598+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:51.274598+0000","last_undegraded":"2021-01-15T18:50:51.274598+0000","last_fullsized":"2021-01-15T18:50:51.274598+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9279,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1491110,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9289","reported_seq":"12234","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:49.447192+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:49.447192+0000","last_peered":"2021-01-15T18:50:49.447192+0000","last_clean":"2021-01-15T18:50:49.447192+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:49.447192+0000","last_undegraded":"2021-01-15T18:50:49.447192+0000","last_fullsized":"2021-01-15T18:50:49.447192+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9289,"ondisk_log_size":9289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1658987,"num_objects":204,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9288,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9601","reported_seq":"9861","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:51.285937+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:51.285937+0000","last_peered":"2021-01-15T18:50:51.285937+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:51.285937+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9601,"ondisk_log_size":9601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1564672,"num_objects":191,"num_object_clones":0,"num_object_copies":764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9648","reported_seq":"10253","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.385870+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:51.385870+0000","last_peered":"2021-01-15T18:50:51.385870+0000","last_clean":"2021-01-15T18:50:51.385870+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:51.385870+0000","last_undegraded":"2021-01-15T18:50:51.385870+0000","last_fullsized":"2021-01-15T18:50:51.385870+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9648,"ondisk_log_size":9648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1474726,"num_objects":181,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9646,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9356","reported_seq":"9581","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.310917+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:51.310917+0000","last_peered":"2021-01-15T18:50:51.310917+0000","last_clean":"2021-01-15T18:50:51.310917+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:51.310917+0000","last_undegraded":"2021-01-15T18:50:51.310917+0000","last_fullsized":"2021-01-15T18:50:51.310917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9356,"ondisk_log_size":9356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1392806,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9503","reported_seq":"12378","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.125846+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:51.125846+0000","last_peered":"2021-01-15T18:50:51.125846+0000","last_clean":"2021-01-15T18:50:51.125846+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:51.125846+0000","last_undegraded":"2021-01-15T18:50:51.125846+0000","last_fullsized":"2021-01-15T18:50:51.125846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9503,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1818790,"num_objects":223,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9663","reported_seq":"12673","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.051233+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:51.051233+0000","last_peered":"2021-01-15T18:50:51.051233+0000","last_clean":"2021-01-15T18:50:51.051233+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:51.051233+0000","last_undegraded":"2021-01-15T18:50:51.051233+0000","last_fullsized":"2021-01-15T18:50:51.051233+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9663,"ondisk_log_size":9663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1585235,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9545","reported_seq":"12494","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.132013+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:51.132013+0000","last_peered":"2021-01-15T18:50:51.132013+0000","last_clean":"2021-01-15T18:50:51.132013+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:51.132013+0000","last_undegraded":"2021-01-15T18:50:51.132013+0000","last_fullsized":"2021-01-15T18:50:51.132013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9545,"ondisk_log_size":9545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1503315,"num_objects":184,"num_object_clones":0,"num_object_copies":736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9577","reported_seq":"12531","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.263292+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:51.263292+0000","last_peered":"2021-01-15T18:50:51.263292+0000","last_clean":"2021-01-15T18:50:51.263292+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:51.263292+0000","last_undegraded":"2021-01-15T18:50:51.263292+0000","last_fullsized":"2021-01-15T18:50:51.263292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9577,"ondisk_log_size":9577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1478905,"num_objects":182,"num_object_clones":0,"num_object_copies":728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9574,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9371","reported_seq":"12224","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.051699+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:51.051699+0000","last_peered":"2021-01-15T18:50:51.051699+0000","last_clean":"2021-01-15T18:50:51.051699+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:51.051699+0000","last_undegraded":"2021-01-15T18:50:51.051699+0000","last_fullsized":"2021-01-15T18:50:51.051699+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9371,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1499302,"num_objects":184,"num_object_clones":0,"num_object_copies":736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9395","reported_seq":"12238","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.393625+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:51.393625+0000","last_peered":"2021-01-15T18:50:51.393625+0000","last_clean":"2021-01-15T18:50:51.393625+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:51.393625+0000","last_undegraded":"2021-01-15T18:50:51.393625+0000","last_fullsized":"2021-01-15T18:50:51.393625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9395,"ondisk_log_size":9395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1409190,"num_objects":173,"num_object_clones":0,"num_object_copies":692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9302","reported_seq":"12211","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:49.459913+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:49.459913+0000","last_peered":"2021-01-15T18:50:49.459913+0000","last_clean":"2021-01-15T18:50:49.459913+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:49.459913+0000","last_undegraded":"2021-01-15T18:50:49.459913+0000","last_fullsized":"2021-01-15T18:50:49.459913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9302,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1417382,"num_objects":174,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9682","reported_seq":"9921","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:51.404687+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:51.404687+0000","last_peered":"2021-01-15T18:50:51.404687+0000","last_clean":"2021-01-15T18:50:51.404687+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:51.404687+0000","last_undegraded":"2021-01-15T18:50:51.404687+0000","last_fullsized":"2021-01-15T18:50:51.404687+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9682,"ondisk_log_size":9682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1671334,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9680,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9872","reported_seq":"11415","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:51.250673+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:51.250673+0000","last_peered":"2021-01-15T18:50:51.250673+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:51.250673+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9872,"ondisk_log_size":9872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1392972,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9358","reported_seq":"12236","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:49.264056+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:49.264056+0000","last_peered":"2021-01-15T18:50:49.264056+0000","last_clean":"2021-01-15T18:50:49.264056+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:49.264056+0000","last_undegraded":"2021-01-15T18:50:49.264056+0000","last_fullsized":"2021-01-15T18:50:49.264056+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9358,"ondisk_log_size":9358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1482918,"num_objects":182,"num_object_clones":0,"num_object_copies":728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":24488236,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151721,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18066,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":50438144,"data_stored":50438144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151748,"ondisk_log_size":151748,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215240,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313320,"kb_used_data":12912,"kb_used_omap":0,"kb_used_meta":300351,"kb_avail":94058520,"statfs":{"total":96636764160,"available":96315924480,"internally_reserved":0,"allocated":13221888,"data_stored":12938683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":307559476},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":6.3209999999999997},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.501}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":6.2249999999999996},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":7.5049999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":7.5460000000000003},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":6.6079999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":13960,"kb_used_omap":0,"kb_used_meta":283648,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":14295040,"data_stored":14011835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":290455552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":11.710000000000001},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":2.4420000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":11.827999999999999},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":2.6419999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":2.5459999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297292,"kb_used_data":13508,"kb_used_omap":0,"kb_used_meta":283775,"kb_avail":94074548,"statfs":{"total":96636764160,"available":96332337152,"internally_reserved":0,"allocated":13832192,"data_stored":13548987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":290585652},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":1.218},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.3109999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":6.5780000000000003},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":16.359000000000002},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":16.472999999999999}]}]},{"osd":1,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":12812,"kb_used_omap":0,"kb_used_meta":295039,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":13119488,"data_stored":12836283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":302119988},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":3.9849999999999999,"15min":3.7349999999999999},"min":{"1min":0.377,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":50.558999999999997,"5min":50.558999999999997,"15min":50.558999999999997},"last":3.9689999999999999},{"interface":"front","average":{"1min":4.7050000000000001,"5min":2.2450000000000001,"15min":2.246},"min":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":49.470999999999997,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.98299999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":4.5430000000000001,"5min":2.8140000000000001,"15min":2.8439999999999999},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":17.523,"5min":24.457999999999998,"15min":24.457999999999998},"last":4.0369999999999999},{"interface":"front","average":{"1min":3.081,"5min":2.3260000000000001,"15min":2.044},"min":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":24.795000000000002,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.89100000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:49:49 2021","interfaces":[{"interface":"back","average":{"1min":3.1019999999999999,"5min":2.75,"15min":3.601},"min":{"1min":0.55700000000000005,"5min":0.498,"15min":0.498},"max":{"1min":24.582999999999998,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":3.3849999999999998,"5min":2.9990000000000001,"15min":2.7109999999999999},"min":{"1min":0.66400000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":24.696000000000002,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.1499999999999999}]}]}]}} 2021-01-15T18:50:56.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:50:57.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:57.453 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:50:57.490 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2021-01-15T18:50:56.693189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16697644,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152672,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18100,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152699,"ondisk_log_size":152699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1203448,"kb_used_data":37592,"kb_used_omap":2,"kb_used_meta":1165693,"kb_avail":376283912,"statfs":{"total":386547056640,"available":385314725888,"internally_reserved":0,"allocated":38494208,"data_stored":37361388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1193669788},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,4,4,16,12,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":137,"apply_latency_ms":137,"commit_latency_ns":137000000,"apply_latency_ns":137000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21725184,"num_objects":-2652,"num_object_clones":0,"num_object_copies":-10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2652,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007380"},"pg_stats":[{"pgid":"1.f","version":"54'9380","reported_seq":"9652","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.240937+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:56.240937+0000","last_peered":"2021-01-15T18:50:56.240937+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:56.240937+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9380,"ondisk_log_size":9380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1048576,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9339","reported_seq":"12277","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.136976+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:56.136976+0000","last_peered":"2021-01-15T18:50:56.136976+0000","last_clean":"2021-01-15T18:50:56.136976+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:56.136976+0000","last_undegraded":"2021-01-15T18:50:56.136976+0000","last_fullsized":"2021-01-15T18:50:56.136976+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9339,"ondisk_log_size":9339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9347","reported_seq":"12292","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:54.126342+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:54.126342+0000","last_peered":"2021-01-15T18:50:54.126342+0000","last_clean":"2021-01-15T18:50:54.126342+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:54.126342+0000","last_undegraded":"2021-01-15T18:50:54.126342+0000","last_fullsized":"2021-01-15T18:50:54.126342+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9347,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1183851,"num_objects":146,"num_object_clones":0,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9346,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9667","reported_seq":"9927","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.329839+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:56.329839+0000","last_peered":"2021-01-15T18:50:56.329839+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:56.329839+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9667,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9697","reported_seq":"10303","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.334282+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:56.334282+0000","last_peered":"2021-01-15T18:50:56.334282+0000","last_clean":"2021-01-15T18:50:56.334282+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:56.334282+0000","last_undegraded":"2021-01-15T18:50:56.334282+0000","last_fullsized":"2021-01-15T18:50:56.334282+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9697,"ondisk_log_size":9697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1073318,"num_objects":132,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9404","reported_seq":"9629","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.414805+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:56.414805+0000","last_peered":"2021-01-15T18:50:56.414805+0000","last_clean":"2021-01-15T18:50:56.414805+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:56.414805+0000","last_undegraded":"2021-01-15T18:50:56.414805+0000","last_fullsized":"2021-01-15T18:50:56.414805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9404,"ondisk_log_size":9404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9585","reported_seq":"12458","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.398652+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:56.398652+0000","last_peered":"2021-01-15T18:50:56.398652+0000","last_clean":"2021-01-15T18:50:56.398652+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:56.398652+0000","last_undegraded":"2021-01-15T18:50:56.398652+0000","last_fullsized":"2021-01-15T18:50:56.398652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9585,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1147046,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9717","reported_seq":"12728","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.358961+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:56.358961+0000","last_peered":"2021-01-15T18:50:56.358961+0000","last_clean":"2021-01-15T18:50:56.358961+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:56.358961+0000","last_undegraded":"2021-01-15T18:50:56.358961+0000","last_fullsized":"2021-01-15T18:50:56.358961+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9717,"ondisk_log_size":9717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1142867,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9612","reported_seq":"12560","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.488900+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:56.488900+0000","last_peered":"2021-01-15T18:50:56.488900+0000","last_clean":"2021-01-15T18:50:56.488900+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:56.488900+0000","last_undegraded":"2021-01-15T18:50:56.488900+0000","last_fullsized":"2021-01-15T18:50:56.488900+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9612,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954451,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9611,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9642","reported_seq":"12597","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.331552+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:56.331552+0000","last_peered":"2021-01-15T18:50:56.331552+0000","last_clean":"2021-01-15T18:50:56.331552+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:56.331552+0000","last_undegraded":"2021-01-15T18:50:56.331552+0000","last_fullsized":"2021-01-15T18:50:56.331552+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9642,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":946425,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9432","reported_seq":"12283","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.314773+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:56.314773+0000","last_peered":"2021-01-15T18:50:56.314773+0000","last_clean":"2021-01-15T18:50:56.314773+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:56.314773+0000","last_undegraded":"2021-01-15T18:50:56.314773+0000","last_fullsized":"2021-01-15T18:50:56.314773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9432,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9440","reported_seq":"12283","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.414629+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:56.414629+0000","last_peered":"2021-01-15T18:50:56.414629+0000","last_clean":"2021-01-15T18:50:56.414629+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:56.414629+0000","last_undegraded":"2021-01-15T18:50:56.414629+0000","last_fullsized":"2021-01-15T18:50:56.414629+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9440,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1040550,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9351","reported_seq":"12260","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:55.044076+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:55.044076+0000","last_peered":"2021-01-15T18:50:55.044076+0000","last_clean":"2021-01-15T18:50:55.044076+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:55.044076+0000","last_undegraded":"2021-01-15T18:50:55.044076+0000","last_fullsized":"2021-01-15T18:50:55.044076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9351,"ondisk_log_size":9351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1015974,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9743","reported_seq":"9982","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.405715+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:56.405715+0000","last_peered":"2021-01-15T18:50:56.405715+0000","last_clean":"2021-01-15T18:50:56.405715+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:56.405715+0000","last_undegraded":"2021-01-15T18:50:56.405715+0000","last_fullsized":"2021-01-15T18:50:56.405715+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9743,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1171622,"num_objects":144,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9929","reported_seq":"11540","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.332320+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:56.332320+0000","last_peered":"2021-01-15T18:50:56.332320+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:56.332320+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9929,"ondisk_log_size":9929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":926028,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9414","reported_seq":"12293","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:54.349205+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:54.349205+0000","last_peered":"2021-01-15T18:50:54.349205+0000","last_clean":"2021-01-15T18:50:54.349205+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:54.349205+0000","last_undegraded":"2021-01-15T18:50:54.349205+0000","last_fullsized":"2021-01-15T18:50:54.349205+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9414,"ondisk_log_size":9414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024166,"num_objects":126,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":16697644,"num_objects":2053,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152672,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18100,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35102720,"data_stored":35102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152699,"ondisk_log_size":152699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":8468,"kb_used_omap":0,"kb_used_meta":294079,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":8671232,"data_stored":8388027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":301136948},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,5,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.98099999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":31.707999999999998},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":26.864999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.9339999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.9990000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303796,"kb_used_data":10240,"kb_used_omap":0,"kb_used_meta":293504,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":10485760,"data_stored":10202555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":300548096},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":3.8919999999999999},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.505}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":4},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":18.498000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":20.838000000000001},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":20.739999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303508,"kb_used_data":9836,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94068332,"statfs":{"total":96636764160,"available":96325971968,"internally_reserved":0,"allocated":10072064,"data_stored":9788859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,8,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":1.6970000000000001},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":1.903}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":0.51400000000000001},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":1.103}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.417}]}]},{"osd":1,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293540,"kb_used_data":9048,"kb_used_omap":0,"kb_used_meta":284479,"kb_avail":94078300,"statfs":{"total":96636764160,"available":96336179200,"internally_reserved":0,"allocated":9265152,"data_stored":8981947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291306548},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.84499999999999997}]}]}]}} 2021-01-15T18:50:57.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:50:59.168 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:50:59.168 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3307/8212 objects degraded (40.270%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2053,"data_bytes":16697644,"bytes_used":1232330752,"bytes_avail":385314725888,"bytes_total":386547056640,"degraded_objects":3307,"degraded_total":8212,"degraded_ratio":0.4027033609352168,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":156},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:50:59.215 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:50:59.215 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:00.784 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:00.821 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2021-01-15T18:51:00.694006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15542572,"num_objects":1912,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152813,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18100,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152840,"ondisk_log_size":152840,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1195220,"kb_used_data":30232,"kb_used_omap":2,"kb_used_meta":1164861,"kb_avail":376292140,"statfs":{"total":386547056640,"available":385323151360,"internally_reserved":0,"allocated":30957568,"data_stored":29824748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1192817820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,7,14,23,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":138,"apply_latency_ms":138,"commit_latency_ns":138000000,"apply_latency_ns":138000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16506880,"num_objects":-2015,"num_object_clones":0,"num_object_copies":-8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007369"},"pg_stats":[{"pgid":"1.f","version":"54'9380","reported_seq":"9652","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.240937+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:50:56.240937+0000","last_peered":"2021-01-15T18:50:56.240937+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:50:56.240937+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9380,"ondisk_log_size":9380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1048576,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9339","reported_seq":"12277","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.136976+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:50:56.136976+0000","last_peered":"2021-01-15T18:50:56.136976+0000","last_clean":"2021-01-15T18:50:56.136976+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:50:56.136976+0000","last_undegraded":"2021-01-15T18:50:56.136976+0000","last_fullsized":"2021-01-15T18:50:56.136976+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9339,"ondisk_log_size":9339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9399","reported_seq":"12344","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:59.117569+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:50:59.117569+0000","last_peered":"2021-01-15T18:50:59.117569+0000","last_clean":"2021-01-15T18:50:59.117569+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:50:59.117569+0000","last_undegraded":"2021-01-15T18:50:59.117569+0000","last_fullsized":"2021-01-15T18:50:59.117569+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9399,"ondisk_log_size":9399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":757867,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9398,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9667","reported_seq":"9927","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.329839+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:50:56.329839+0000","last_peered":"2021-01-15T18:50:56.329839+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:50:56.329839+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9667,"ondisk_log_size":9667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024000,"num_objects":125,"num_object_clones":0,"num_object_copies":500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1209}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9697","reported_seq":"10303","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.334282+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:50:56.334282+0000","last_peered":"2021-01-15T18:50:56.334282+0000","last_clean":"2021-01-15T18:50:56.334282+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:50:56.334282+0000","last_undegraded":"2021-01-15T18:50:56.334282+0000","last_fullsized":"2021-01-15T18:50:56.334282+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9697,"ondisk_log_size":9697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1073318,"num_objects":132,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9404","reported_seq":"9629","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.414805+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:50:56.414805+0000","last_peered":"2021-01-15T18:50:56.414805+0000","last_clean":"2021-01-15T18:50:56.414805+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:50:56.414805+0000","last_undegraded":"2021-01-15T18:50:56.414805+0000","last_fullsized":"2021-01-15T18:50:56.414805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9404,"ondisk_log_size":9404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9585","reported_seq":"12458","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.398652+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:50:56.398652+0000","last_peered":"2021-01-15T18:50:56.398652+0000","last_clean":"2021-01-15T18:50:56.398652+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:50:56.398652+0000","last_undegraded":"2021-01-15T18:50:56.398652+0000","last_fullsized":"2021-01-15T18:50:56.398652+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9585,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1147046,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9717","reported_seq":"12728","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.358961+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:50:56.358961+0000","last_peered":"2021-01-15T18:50:56.358961+0000","last_clean":"2021-01-15T18:50:56.358961+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:50:56.358961+0000","last_undegraded":"2021-01-15T18:50:56.358961+0000","last_fullsized":"2021-01-15T18:50:56.358961+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9717,"ondisk_log_size":9717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1142867,"num_objects":140,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9612","reported_seq":"12560","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.488900+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:50:56.488900+0000","last_peered":"2021-01-15T18:50:56.488900+0000","last_clean":"2021-01-15T18:50:56.488900+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:50:56.488900+0000","last_undegraded":"2021-01-15T18:50:56.488900+0000","last_fullsized":"2021-01-15T18:50:56.488900+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9612,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954451,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9611,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9642","reported_seq":"12597","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.331552+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:50:56.331552+0000","last_peered":"2021-01-15T18:50:56.331552+0000","last_clean":"2021-01-15T18:50:56.331552+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:50:56.331552+0000","last_undegraded":"2021-01-15T18:50:56.331552+0000","last_fullsized":"2021-01-15T18:50:56.331552+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9642,"ondisk_log_size":9642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":946425,"num_objects":117,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9639,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9432","reported_seq":"12283","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.314773+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:50:56.314773+0000","last_peered":"2021-01-15T18:50:56.314773+0000","last_clean":"2021-01-15T18:50:56.314773+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:50:56.314773+0000","last_undegraded":"2021-01-15T18:50:56.314773+0000","last_fullsized":"2021-01-15T18:50:56.314773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9432,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":999590,"num_objects":123,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9440","reported_seq":"12283","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.414629+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:50:56.414629+0000","last_peered":"2021-01-15T18:50:56.414629+0000","last_clean":"2021-01-15T18:50:56.414629+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:50:56.414629+0000","last_undegraded":"2021-01-15T18:50:56.414629+0000","last_fullsized":"2021-01-15T18:50:56.414629+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9440,"ondisk_log_size":9440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1040550,"num_objects":128,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9400","reported_seq":"12309","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:59.731696+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:59.731696+0000","last_peered":"2021-01-15T18:50:59.731696+0000","last_clean":"2021-01-15T18:50:59.731696+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:59.731696+0000","last_undegraded":"2021-01-15T18:50:59.731696+0000","last_fullsized":"2021-01-15T18:50:59.731696+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9400,"ondisk_log_size":9400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":614566,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9743","reported_seq":"9982","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:56.405715+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:50:56.405715+0000","last_peered":"2021-01-15T18:50:56.405715+0000","last_clean":"2021-01-15T18:50:56.405715+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:50:56.405715+0000","last_undegraded":"2021-01-15T18:50:56.405715+0000","last_fullsized":"2021-01-15T18:50:56.405715+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9743,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1171622,"num_objects":144,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9929","reported_seq":"11540","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:50:56.332320+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:50:56.332320+0000","last_peered":"2021-01-15T18:50:56.332320+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:50:56.332320+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9929,"ondisk_log_size":9929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":926028,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9454","reported_seq":"12333","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:59.370052+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:50:59.370052+0000","last_peered":"2021-01-15T18:50:59.370052+0000","last_clean":"2021-01-15T18:50:59.370052+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:50:59.370052+0000","last_undegraded":"2021-01-15T18:50:59.370052+0000","last_fullsized":"2021-01-15T18:50:59.370052+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9454,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":696486,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":15542572,"num_objects":1912,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152813,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18100,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27643904,"data_stored":27643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152840,"ondisk_log_size":152840,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302604,"kb_used_data":8468,"kb_used_omap":0,"kb_used_meta":294079,"kb_avail":94069236,"statfs":{"total":96636764160,"available":96326897664,"internally_reserved":0,"allocated":8671232,"data_stored":8388027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":301136948},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,5,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.98099999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":31.707999999999998},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":26.864999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":1.9339999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.9990000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305872,"kb_used_data":6624,"kb_used_omap":0,"kb_used_meta":299200,"kb_avail":94065968,"statfs":{"total":96636764160,"available":96323551232,"internally_reserved":0,"allocated":6782976,"data_stored":6499771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":306380800},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.101,"5min":2.3410000000000002,"15min":3.3050000000000002},"min":{"1min":0.40600000000000003,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":14.141,"5min":27.870000000000001,"15min":36.380000000000003},"last":2.7759999999999998},{"interface":"front","average":{"1min":1.333,"5min":2.1099999999999999,"15min":1.9490000000000001},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.371},"max":{"1min":5.7140000000000004,"5min":27.861999999999998,"15min":27.861999999999998},"last":7.2889999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":2.673},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":8.4079999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":5.2359999999999998,"5min":3.4809999999999999,"15min":2.944},"min":{"1min":0.625,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":26.981000000000002,"5min":26.981000000000002,"15min":26.981000000000002},"last":7.1920000000000002},{"interface":"front","average":{"1min":2.1019999999999999,"5min":2.254,"15min":2.5550000000000002},"min":{"1min":0.63,"5min":0.52400000000000002,"15min":0.46300000000000002},"max":{"1min":6.5510000000000002,"5min":16.821999999999999,"15min":18.34},"last":7.3899999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293204,"kb_used_data":6092,"kb_used_omap":0,"kb_used_meta":287103,"kb_avail":94078636,"statfs":{"total":96636764160,"available":96336523264,"internally_reserved":0,"allocated":6238208,"data_stored":5955003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":293993524},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":12.327999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":12.419}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":4.7409999999999997},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":3.702},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.008}]}]},{"osd":1,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293540,"kb_used_data":9048,"kb_used_omap":0,"kb_used_meta":284479,"kb_avail":94078300,"statfs":{"total":96636764160,"available":96336179200,"internally_reserved":0,"allocated":9265152,"data_stored":8981947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":291306548},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.84499999999999997}]}]}]}} 2021-01-15T18:51:03.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:04.870 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:04.887 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:04.906 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2021-01-15T18:51:04.695807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8489260,"num_objects":1051,"num_object_clones":0,"num_object_copies":4204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153674,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18143,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153701,"ondisk_log_size":153701,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1193832,"kb_used_data":20028,"kb_used_omap":2,"kb_used_meta":1173629,"kb_avail":376293528,"statfs":{"total":386547056640,"available":385324572672,"internally_reserved":0,"allocated":20508672,"data_stored":19375852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1201796252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,9,5,20,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":137,"apply_latency_ms":137,"commit_latency_ns":137000000,"apply_latency_ns":137000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15998976,"num_objects":-1953,"num_object_clones":0,"num_object_copies":-7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004971"},"pg_stats":[{"pgid":"1.f","version":"54'9427","reported_seq":"9699","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:01.222168+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:01.222168+0000","last_peered":"2021-01-15T18:51:01.222168+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:01.222168+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9427,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":663552,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9399","reported_seq":"12337","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.180295+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:01.180295+0000","last_peered":"2021-01-15T18:51:01.180295+0000","last_clean":"2021-01-15T18:51:01.180295+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:01.180295+0000","last_undegraded":"2021-01-15T18:51:01.180295+0000","last_fullsized":"2021-01-15T18:51:01.180295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9399,"ondisk_log_size":9399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":508070,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9397,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9462","reported_seq":"12406","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:04.520070+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:04.520070+0000","last_peered":"2021-01-15T18:51:04.520070+0000","last_clean":"2021-01-15T18:51:04.520070+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:04.520070+0000","last_undegraded":"2021-01-15T18:51:04.520070+0000","last_fullsized":"2021-01-15T18:51:04.520070+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9462,"ondisk_log_size":9462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241771,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9461,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9711","reported_seq":"9971","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:01.142369+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:01.142369+0000","last_peered":"2021-01-15T18:51:01.142369+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:01.142369+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9711,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":663552,"num_objects":81,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9711,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9757","reported_seq":"10363","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.409373+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:01.409373+0000","last_peered":"2021-01-15T18:51:01.409373+0000","last_clean":"2021-01-15T18:51:01.409373+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:01.409373+0000","last_undegraded":"2021-01-15T18:51:01.409373+0000","last_fullsized":"2021-01-15T18:51:01.409373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9757,"ondisk_log_size":9757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":581798,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9458","reported_seq":"9682","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:00.879659+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:00.879659+0000","last_peered":"2021-01-15T18:51:00.879659+0000","last_clean":"2021-01-15T18:51:00.879659+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:00.879659+0000","last_undegraded":"2021-01-15T18:51:00.879659+0000","last_fullsized":"2021-01-15T18:51:00.879659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9458,"ondisk_log_size":9458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":557222,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9456,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9652","reported_seq":"12526","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.449148+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:01.449148+0000","last_peered":"2021-01-15T18:51:01.449148+0000","last_clean":"2021-01-15T18:51:01.449148+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:01.449148+0000","last_undegraded":"2021-01-15T18:51:01.449148+0000","last_fullsized":"2021-01-15T18:51:01.449148+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9652,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598182,"num_objects":74,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9781","reported_seq":"12792","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.225761+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:01.225761+0000","last_peered":"2021-01-15T18:51:01.225761+0000","last_clean":"2021-01-15T18:51:01.225761+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:01.225761+0000","last_undegraded":"2021-01-15T18:51:01.225761+0000","last_fullsized":"2021-01-15T18:51:01.225761+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9781,"ondisk_log_size":9781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":618579,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9668","reported_seq":"12616","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.496352+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:01.496352+0000","last_peered":"2021-01-15T18:51:01.496352+0000","last_clean":"2021-01-15T18:51:01.496352+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:01.496352+0000","last_undegraded":"2021-01-15T18:51:01.496352+0000","last_fullsized":"2021-01-15T18:51:01.496352+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9668,"ondisk_log_size":9668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":495699,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9697","reported_seq":"12652","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:00.962384+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:00.962384+0000","last_peered":"2021-01-15T18:51:00.962384+0000","last_clean":"2021-01-15T18:51:00.962384+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:00.962384+0000","last_undegraded":"2021-01-15T18:51:00.962384+0000","last_fullsized":"2021-01-15T18:51:00.962384+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9697,"ondisk_log_size":9697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":495865,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9694,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9495","reported_seq":"12348","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.290533+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:01.290533+0000","last_peered":"2021-01-15T18:51:01.290533+0000","last_clean":"2021-01-15T18:51:01.290533+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:01.290533+0000","last_undegraded":"2021-01-15T18:51:01.290533+0000","last_fullsized":"2021-01-15T18:51:01.290533+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9495,"ondisk_log_size":9495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":483494,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9493,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9504","reported_seq":"12347","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.460082+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:01.460082+0000","last_peered":"2021-01-15T18:51:01.460082+0000","last_clean":"2021-01-15T18:51:01.460082+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:01.460082+0000","last_undegraded":"2021-01-15T18:51:01.460082+0000","last_fullsized":"2021-01-15T18:51:01.460082+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9504,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":516262,"num_objects":64,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9400","reported_seq":"12309","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:50:59.731696+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:50:59.731696+0000","last_peered":"2021-01-15T18:50:59.731696+0000","last_clean":"2021-01-15T18:50:59.731696+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:50:59.731696+0000","last_undegraded":"2021-01-15T18:50:59.731696+0000","last_fullsized":"2021-01-15T18:50:59.731696+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9400,"ondisk_log_size":9400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":614566,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9812","reported_seq":"10049","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:01.458660+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:01.458660+0000","last_peered":"2021-01-15T18:51:01.458660+0000","last_clean":"2021-01-15T18:51:01.458660+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:01.458660+0000","last_undegraded":"2021-01-15T18:51:01.458660+0000","last_fullsized":"2021-01-15T18:51:01.458660+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9812,"ondisk_log_size":9812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":606374,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'9981","reported_seq":"11678","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:01.373340+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:01.373340+0000","last_peered":"2021-01-15T18:51:01.373340+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:01.373340+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9981,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":500044,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9978,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9497","reported_seq":"12376","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:04.477401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:04.477401+0000","last_peered":"2021-01-15T18:51:04.477401+0000","last_clean":"2021-01-15T18:51:04.477401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:04.477401+0000","last_undegraded":"2021-01-15T18:51:04.477401+0000","last_fullsized":"2021-01-15T18:51:04.477401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9497,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":344230,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9495,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8489260,"num_objects":1051,"num_object_clones":0,"num_object_copies":4204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153674,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18143,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16691200,"data_stored":16691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153701,"ondisk_log_size":153701,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215242,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305104,"kb_used_data":5524,"kb_used_omap":0,"kb_used_meta":299519,"kb_avail":94066736,"statfs":{"total":96636764160,"available":96324337664,"internally_reserved":0,"allocated":5656576,"data_stored":5373371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":306707508},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":5.6900000000000004},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":5.5970000000000004},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":6.9320000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":8.8249999999999993}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295780,"kb_used_data":3124,"kb_used_omap":0,"kb_used_meta":292608,"kb_avail":94076060,"statfs":{"total":96636764160,"available":96333885440,"internally_reserved":0,"allocated":3198976,"data_stored":2915771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":299630592},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":5.0629999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.754},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":8.5}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":10.053000000000001},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":10.156000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293204,"kb_used_data":6092,"kb_used_omap":0,"kb_used_meta":287103,"kb_avail":94078636,"statfs":{"total":96636764160,"available":96336523264,"internally_reserved":0,"allocated":6238208,"data_stored":5955003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":293993524},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":12.327999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":12.419}]},{"osd":2,"last update":"Fri Jan 15 18:50:02 2021","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":3.6749999999999998,"15min":3.04},"min":{"1min":0.46800000000000003,"5min":0.434,"15min":0.39100000000000001},"max":{"1min":16.888000000000002,"5min":25.239999999999998,"15min":27.77},"last":4.7409999999999997},{"interface":"front","average":{"1min":3.6499999999999999,"5min":2.6120000000000001,"15min":2.7120000000000002},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":21.614999999999998,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":3.702},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":1.008}]}]},{"osd":1,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299744,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":294399,"kb_avail":94072096,"statfs":{"total":96636764160,"available":96329826304,"internally_reserved":0,"allocated":5414912,"data_stored":5131707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":301464628},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":4.5780000000000003},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":5.1470000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":28.584},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":11.023999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":5.258},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":5.3419999999999996}]}]}]}} 2021-01-15T18:51:04.908 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:06.359 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3264/4204 objects degraded (77.640%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":716,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1051,"data_bytes":8489260,"bytes_used":1222483968,"bytes_avail":385324572672,"bytes_total":386547056640,"degraded_objects":3264,"degraded_total":4204,"degraded_ratio":0.77640342530922934,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:06.392 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:06.393 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:07.541 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:07.554 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:07.596 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2021-01-15T18:51:06.698807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2238764,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154437,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154464,"ondisk_log_size":154464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1179656,"kb_used_data":9040,"kb_used_omap":2,"kb_used_meta":1170429,"kb_avail":376307704,"statfs":{"total":386547056640,"available":385339088896,"internally_reserved":0,"allocated":9256960,"data_stored":8124140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1198519452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,8,9,17,18,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":131,"apply_latency_ms":131,"commit_latency_ns":131000000,"apply_latency_ns":131000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21315584,"num_objects":-2602,"num_object_clones":0,"num_object_copies":-10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006003"},"pg_stats":[{"pgid":"1.f","version":"54'9494","reported_seq":"9765","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.398090+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:06.398090+0000","last_peered":"2021-01-15T18:51:06.398090+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:06.398090+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9494,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9451","reported_seq":"12389","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.484876+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:06.484876+0000","last_peered":"2021-01-15T18:51:06.484876+0000","last_clean":"2021-01-15T18:51:06.484876+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:06.484876+0000","last_undegraded":"2021-01-15T18:51:06.484876+0000","last_fullsized":"2021-01-15T18:51:06.484876+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9451,"ondisk_log_size":9451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82086,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9449,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9462","reported_seq":"12406","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:04.520070+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:04.520070+0000","last_peered":"2021-01-15T18:51:04.520070+0000","last_clean":"2021-01-15T18:51:04.520070+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:04.520070+0000","last_undegraded":"2021-01-15T18:51:04.520070+0000","last_fullsized":"2021-01-15T18:51:04.520070+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9462,"ondisk_log_size":9462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241771,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9461,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9778","reported_seq":"10036","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.328703+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:06.328703+0000","last_peered":"2021-01-15T18:51:06.328703+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:06.328703+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9778,"ondisk_log_size":9778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9778,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9811","reported_seq":"10417","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.467075+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:06.467075+0000","last_peered":"2021-01-15T18:51:06.467075+0000","last_clean":"2021-01-15T18:51:06.467075+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:06.467075+0000","last_undegraded":"2021-01-15T18:51:06.467075+0000","last_fullsized":"2021-01-15T18:51:06.467075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9811,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139430,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9513","reported_seq":"9737","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.226812+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:06.226812+0000","last_peered":"2021-01-15T18:51:06.226812+0000","last_clean":"2021-01-15T18:51:06.226812+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:06.226812+0000","last_undegraded":"2021-01-15T18:51:06.226812+0000","last_fullsized":"2021-01-15T18:51:06.226812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9513,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9711","reported_seq":"12586","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.122589+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:06.122589+0000","last_peered":"2021-01-15T18:51:06.122589+0000","last_clean":"2021-01-15T18:51:06.122589+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:06.122589+0000","last_undegraded":"2021-01-15T18:51:06.122589+0000","last_fullsized":"2021-01-15T18:51:06.122589+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9711,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114854,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9839","reported_seq":"12850","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.354879+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:06.354879+0000","last_peered":"2021-01-15T18:51:06.354879+0000","last_clean":"2021-01-15T18:51:06.354879+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:06.354879+0000","last_undegraded":"2021-01-15T18:51:06.354879+0000","last_fullsized":"2021-01-15T18:51:06.354879+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9839,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143443,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9713","reported_seq":"12662","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.465547+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:06.465547+0000","last_peered":"2021-01-15T18:51:06.465547+0000","last_clean":"2021-01-15T18:51:06.465547+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:06.465547+0000","last_undegraded":"2021-01-15T18:51:06.465547+0000","last_fullsized":"2021-01-15T18:51:06.465547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9713,"ondisk_log_size":9713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":127059,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9747","reported_seq":"12701","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.359906+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:06.359906+0000","last_peered":"2021-01-15T18:51:06.359906+0000","last_clean":"2021-01-15T18:51:06.359906+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:06.359906+0000","last_undegraded":"2021-01-15T18:51:06.359906+0000","last_fullsized":"2021-01-15T18:51:06.359906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9747,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":86265,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9541","reported_seq":"12394","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.089303+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:06.089303+0000","last_peered":"2021-01-15T18:51:06.089303+0000","last_clean":"2021-01-15T18:51:06.089303+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:06.089303+0000","last_undegraded":"2021-01-15T18:51:06.089303+0000","last_fullsized":"2021-01-15T18:51:06.089303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9541,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9555","reported_seq":"12398","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.119065+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:06.119065+0000","last_peered":"2021-01-15T18:51:06.119065+0000","last_clean":"2021-01-15T18:51:06.119065+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:06.119065+0000","last_undegraded":"2021-01-15T18:51:06.119065+0000","last_fullsized":"2021-01-15T18:51:06.119065+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9555,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98470,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9454","reported_seq":"12363","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:04.975821+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:04.975821+0000","last_peered":"2021-01-15T18:51:04.975821+0000","last_clean":"2021-01-15T18:51:04.975821+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:04.975821+0000","last_undegraded":"2021-01-15T18:51:04.975821+0000","last_fullsized":"2021-01-15T18:51:04.975821+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9454,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":172198,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9870","reported_seq":"10110","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.389551+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:06.389551+0000","last_peered":"2021-01-15T18:51:06.389551+0000","last_clean":"2021-01-15T18:51:06.389551+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:06.389551+0000","last_undegraded":"2021-01-15T18:51:06.389551+0000","last_fullsized":"2021-01-15T18:51:06.389551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9870,"ondisk_log_size":9870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131238,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9868,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10028","reported_seq":"11813","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.296163+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:06.296163+0000","last_peered":"2021-01-15T18:51:06.296163+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:06.296163+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10028,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115020,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9497","reported_seq":"12376","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:04.477401+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:04.477401+0000","last_peered":"2021-01-15T18:51:04.477401+0000","last_clean":"2021-01-15T18:51:04.477401+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:04.477401+0000","last_undegraded":"2021-01-15T18:51:04.477401+0000","last_fullsized":"2021-01-15T18:51:04.477401+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9497,"ondisk_log_size":9497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":344230,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9495,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":2238764,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154437,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6057984,"data_stored":6057984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154464,"ondisk_log_size":154464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294644,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":293183,"kb_avail":94077196,"statfs":{"total":96636764160,"available":96335048704,"internally_reserved":0,"allocated":1437696,"data_stored":1154491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300219444},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":5.6900000000000004},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":4.077},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":18.341000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4039999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.8700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295780,"kb_used_data":3124,"kb_used_omap":0,"kb_used_meta":292608,"kb_avail":94076060,"statfs":{"total":96636764160,"available":96333885440,"internally_reserved":0,"allocated":3198976,"data_stored":2915771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":299630592},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":5.0629999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.754},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":8.5}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":10.053000000000001},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":10.156000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299540,"kb_used_data":2668,"kb_used_omap":0,"kb_used_meta":296831,"kb_avail":94072300,"statfs":{"total":96636764160,"available":96330035200,"internally_reserved":0,"allocated":2732032,"data_stored":2448827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":303954996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":8.9320000000000004},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":9.0449999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":3.6280000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:50:08 2021","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":4.4329999999999998,"15min":4.2210000000000001},"min":{"1min":0.63,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":6.8470000000000004,"5min":56.615000000000002,"15min":56.615000000000002},"last":3.5230000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":3.1890000000000001,"15min":3.3330000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":6.5030000000000001,"5min":24.734999999999999,"15min":56.927},"last":0.64500000000000002}]}]},{"osd":1,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289692,"kb_used_data":1844,"kb_used_omap":0,"kb_used_meta":287807,"kb_avail":94082148,"statfs":{"total":96636764160,"available":96340119552,"internally_reserved":0,"allocated":1888256,"data_stored":1605051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":294714420},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":10.411},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":2.262}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.256}]}]}]}} 2021-01-15T18:51:07.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Removed 77320 objects 2021-01-15T18:51:07.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi148.stdout:Clean up completed and total clean up time :427.54 2021-01-15T18:51:10.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:10.910 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:10.912 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:10.923 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2021-01-15T18:51:10.699677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1501460,"num_objects":197,"num_object_clones":0,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154528,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154555,"ondisk_log_size":154555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189144,"kb_used_data":4136,"kb_used_omap":2,"kb_used_meta":1184893,"kb_avail":376298216,"statfs":{"total":386547056640,"available":385329373184,"internally_reserved":0,"allocated":4235264,"data_stored":3102444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1213330588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,9,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15196184,"num_objects":-1856,"num_object_clones":0,"num_object_copies":-7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006049"},"pg_stats":[{"pgid":"1.f","version":"54'9494","reported_seq":"9765","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.398090+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:06.398090+0000","last_peered":"2021-01-15T18:51:06.398090+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:06.398090+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9494,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9451","reported_seq":"12389","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.484876+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:06.484876+0000","last_peered":"2021-01-15T18:51:06.484876+0000","last_clean":"2021-01-15T18:51:06.484876+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:06.484876+0000","last_undegraded":"2021-01-15T18:51:06.484876+0000","last_fullsized":"2021-01-15T18:51:06.484876+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9451,"ondisk_log_size":9451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82086,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9449,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12437","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.556595+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.556595+0000","last_peered":"2021-01-15T18:51:07.556595+0000","last_clean":"2021-01-15T18:51:07.556595+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.556595+0000","last_undegraded":"2021-01-15T18:51:07.556595+0000","last_fullsized":"2021-01-15T18:51:07.556595+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9778","reported_seq":"10036","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.328703+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:06.328703+0000","last_peered":"2021-01-15T18:51:06.328703+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:06.328703+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9778,"ondisk_log_size":9778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9778,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9811","reported_seq":"10417","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.467075+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:06.467075+0000","last_peered":"2021-01-15T18:51:06.467075+0000","last_clean":"2021-01-15T18:51:06.467075+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:06.467075+0000","last_undegraded":"2021-01-15T18:51:06.467075+0000","last_fullsized":"2021-01-15T18:51:06.467075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9811,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139430,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9513","reported_seq":"9737","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.226812+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:06.226812+0000","last_peered":"2021-01-15T18:51:06.226812+0000","last_clean":"2021-01-15T18:51:06.226812+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:06.226812+0000","last_undegraded":"2021-01-15T18:51:06.226812+0000","last_fullsized":"2021-01-15T18:51:06.226812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9513,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9711","reported_seq":"12586","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.122589+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:06.122589+0000","last_peered":"2021-01-15T18:51:06.122589+0000","last_clean":"2021-01-15T18:51:06.122589+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:06.122589+0000","last_undegraded":"2021-01-15T18:51:06.122589+0000","last_fullsized":"2021-01-15T18:51:06.122589+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9711,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114854,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9839","reported_seq":"12850","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.354879+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:06.354879+0000","last_peered":"2021-01-15T18:51:06.354879+0000","last_clean":"2021-01-15T18:51:06.354879+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:06.354879+0000","last_undegraded":"2021-01-15T18:51:06.354879+0000","last_fullsized":"2021-01-15T18:51:06.354879+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9839,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143443,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9713","reported_seq":"12662","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.465547+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:06.465547+0000","last_peered":"2021-01-15T18:51:06.465547+0000","last_clean":"2021-01-15T18:51:06.465547+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:06.465547+0000","last_undegraded":"2021-01-15T18:51:06.465547+0000","last_fullsized":"2021-01-15T18:51:06.465547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9713,"ondisk_log_size":9713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":127059,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9747","reported_seq":"12701","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.359906+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:06.359906+0000","last_peered":"2021-01-15T18:51:06.359906+0000","last_clean":"2021-01-15T18:51:06.359906+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:06.359906+0000","last_undegraded":"2021-01-15T18:51:06.359906+0000","last_fullsized":"2021-01-15T18:51:06.359906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9747,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":86265,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9541","reported_seq":"12394","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.089303+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:06.089303+0000","last_peered":"2021-01-15T18:51:06.089303+0000","last_clean":"2021-01-15T18:51:06.089303+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:06.089303+0000","last_undegraded":"2021-01-15T18:51:06.089303+0000","last_fullsized":"2021-01-15T18:51:06.089303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9541,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9555","reported_seq":"12398","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.119065+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:06.119065+0000","last_peered":"2021-01-15T18:51:06.119065+0000","last_clean":"2021-01-15T18:51:06.119065+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:06.119065+0000","last_undegraded":"2021-01-15T18:51:06.119065+0000","last_fullsized":"2021-01-15T18:51:06.119065+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9555,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98470,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12383","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.487567+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:07.487567+0000","last_peered":"2021-01-15T18:51:07.487567+0000","last_clean":"2021-01-15T18:51:07.487567+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:07.487567+0000","last_undegraded":"2021-01-15T18:51:07.487567+0000","last_fullsized":"2021-01-15T18:51:07.487567+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9870","reported_seq":"10110","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.389551+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:06.389551+0000","last_peered":"2021-01-15T18:51:06.389551+0000","last_clean":"2021-01-15T18:51:06.389551+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:06.389551+0000","last_undegraded":"2021-01-15T18:51:06.389551+0000","last_fullsized":"2021-01-15T18:51:06.389551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9870,"ondisk_log_size":9870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131238,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9868,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10028","reported_seq":"11813","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.296163+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:06.296163+0000","last_peered":"2021-01-15T18:51:06.296163+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:06.296163+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10028,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115020,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12417","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.615919+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.615919+0000","last_peered":"2021-01-15T18:51:07.615919+0000","last_clean":"2021-01-15T18:51:07.615919+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.615919+0000","last_undegraded":"2021-01-15T18:51:07.615919+0000","last_fullsized":"2021-01-15T18:51:07.615919+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":1501460,"num_objects":197,"num_object_clones":0,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154528,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1855488,"data_stored":1855488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154555,"ondisk_log_size":154555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294644,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":293183,"kb_avail":94077196,"statfs":{"total":96636764160,"available":96335048704,"internally_reserved":0,"allocated":1437696,"data_stored":1154491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300219444},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":5.6900000000000004},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":4.077},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":18.341000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4039999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.8700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.60099999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302340,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069500,"statfs":{"total":96636764160,"available":96327168000,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.59799999999999998},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.62},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.57099999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289692,"kb_used_data":1844,"kb_used_omap":0,"kb_used_meta":287807,"kb_avail":94082148,"statfs":{"total":96636764160,"available":96340119552,"internally_reserved":0,"allocated":1888256,"data_stored":1605051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":294714420},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":10.411},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":2.262}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.256}]}]}]}} 2021-01-15T18:51:10.925 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:11.305 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:11.305 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3220/788 objects degraded (408.629%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":54,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":197,"data_bytes":1501460,"bytes_used":1217683456,"bytes_avail":385329373184,"bytes_total":386547056640,"degraded_objects":3220,"degraded_total":788,"degraded_ratio":4.0862944162436552,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":154},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:11.319 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:11.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:11.610 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:11.612 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:11.623 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2021-01-15T18:51:10.699677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1501460,"num_objects":197,"num_object_clones":0,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154528,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154555,"ondisk_log_size":154555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189144,"kb_used_data":4136,"kb_used_omap":2,"kb_used_meta":1184893,"kb_avail":376298216,"statfs":{"total":386547056640,"available":385329373184,"internally_reserved":0,"allocated":4235264,"data_stored":3102444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1213330588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,9,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":136,"apply_latency_ms":136,"commit_latency_ns":136000000,"apply_latency_ns":136000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15196184,"num_objects":-1856,"num_object_clones":0,"num_object_copies":-7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006049"},"pg_stats":[{"pgid":"1.f","version":"54'9494","reported_seq":"9765","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.398090+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:06.398090+0000","last_peered":"2021-01-15T18:51:06.398090+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:06.398090+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9494,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9494,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9451","reported_seq":"12389","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.484876+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:06.484876+0000","last_peered":"2021-01-15T18:51:06.484876+0000","last_clean":"2021-01-15T18:51:06.484876+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:06.484876+0000","last_undegraded":"2021-01-15T18:51:06.484876+0000","last_fullsized":"2021-01-15T18:51:06.484876+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9451,"ondisk_log_size":9451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82086,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9449,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12437","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.556595+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.556595+0000","last_peered":"2021-01-15T18:51:07.556595+0000","last_clean":"2021-01-15T18:51:07.556595+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.556595+0000","last_undegraded":"2021-01-15T18:51:07.556595+0000","last_fullsized":"2021-01-15T18:51:07.556595+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9778","reported_seq":"10036","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.328703+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:06.328703+0000","last_peered":"2021-01-15T18:51:06.328703+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:06.328703+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9778,"ondisk_log_size":9778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114688,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9778,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9811","reported_seq":"10417","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.467075+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:06.467075+0000","last_peered":"2021-01-15T18:51:06.467075+0000","last_clean":"2021-01-15T18:51:06.467075+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:06.467075+0000","last_undegraded":"2021-01-15T18:51:06.467075+0000","last_fullsized":"2021-01-15T18:51:06.467075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9811,"ondisk_log_size":9811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139430,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9513","reported_seq":"9737","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.226812+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:06.226812+0000","last_peered":"2021-01-15T18:51:06.226812+0000","last_clean":"2021-01-15T18:51:06.226812+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:06.226812+0000","last_undegraded":"2021-01-15T18:51:06.226812+0000","last_fullsized":"2021-01-15T18:51:06.226812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9513,"ondisk_log_size":9513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9711","reported_seq":"12586","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.122589+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:06.122589+0000","last_peered":"2021-01-15T18:51:06.122589+0000","last_clean":"2021-01-15T18:51:06.122589+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:06.122589+0000","last_undegraded":"2021-01-15T18:51:06.122589+0000","last_fullsized":"2021-01-15T18:51:06.122589+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9711,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114854,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9839","reported_seq":"12850","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.354879+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:06.354879+0000","last_peered":"2021-01-15T18:51:06.354879+0000","last_clean":"2021-01-15T18:51:06.354879+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:06.354879+0000","last_undegraded":"2021-01-15T18:51:06.354879+0000","last_fullsized":"2021-01-15T18:51:06.354879+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9839,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143443,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9713","reported_seq":"12662","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.465547+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:06.465547+0000","last_peered":"2021-01-15T18:51:06.465547+0000","last_clean":"2021-01-15T18:51:06.465547+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:06.465547+0000","last_undegraded":"2021-01-15T18:51:06.465547+0000","last_fullsized":"2021-01-15T18:51:06.465547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9713,"ondisk_log_size":9713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":127059,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9712,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9747","reported_seq":"12701","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.359906+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:06.359906+0000","last_peered":"2021-01-15T18:51:06.359906+0000","last_clean":"2021-01-15T18:51:06.359906+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:06.359906+0000","last_undegraded":"2021-01-15T18:51:06.359906+0000","last_fullsized":"2021-01-15T18:51:06.359906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9747,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":86265,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9541","reported_seq":"12394","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.089303+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:06.089303+0000","last_peered":"2021-01-15T18:51:06.089303+0000","last_clean":"2021-01-15T18:51:06.089303+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:06.089303+0000","last_undegraded":"2021-01-15T18:51:06.089303+0000","last_fullsized":"2021-01-15T18:51:06.089303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9541,"ondisk_log_size":9541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106662,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9555","reported_seq":"12398","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.119065+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:06.119065+0000","last_peered":"2021-01-15T18:51:06.119065+0000","last_clean":"2021-01-15T18:51:06.119065+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:06.119065+0000","last_undegraded":"2021-01-15T18:51:06.119065+0000","last_fullsized":"2021-01-15T18:51:06.119065+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9555,"ondisk_log_size":9555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98470,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12383","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.487567+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:07.487567+0000","last_peered":"2021-01-15T18:51:07.487567+0000","last_clean":"2021-01-15T18:51:07.487567+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:07.487567+0000","last_undegraded":"2021-01-15T18:51:07.487567+0000","last_fullsized":"2021-01-15T18:51:07.487567+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9870","reported_seq":"10110","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:06.389551+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:06.389551+0000","last_peered":"2021-01-15T18:51:06.389551+0000","last_clean":"2021-01-15T18:51:06.389551+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:06.389551+0000","last_undegraded":"2021-01-15T18:51:06.389551+0000","last_fullsized":"2021-01-15T18:51:06.389551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9870,"ondisk_log_size":9870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131238,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9868,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10028","reported_seq":"11813","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:06.296163+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:06.296163+0000","last_peered":"2021-01-15T18:51:06.296163+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:06.296163+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10028,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115020,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-620}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12417","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.615919+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.615919+0000","last_peered":"2021-01-15T18:51:07.615919+0000","last_clean":"2021-01-15T18:51:07.615919+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.615919+0000","last_undegraded":"2021-01-15T18:51:07.615919+0000","last_fullsized":"2021-01-15T18:51:07.615919+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":1501460,"num_objects":197,"num_object_clones":0,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154528,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18187,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1855488,"data_stored":1855488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154555,"ondisk_log_size":154555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215243,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294644,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":293183,"kb_avail":94077196,"statfs":{"total":96636764160,"available":96335048704,"internally_reserved":0,"allocated":1437696,"data_stored":1154491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300219444},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":5.6900000000000004},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.628}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":4.077},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":18.341000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":9.4039999999999999},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":3.8700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.60099999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302340,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069500,"statfs":{"total":96636764160,"available":96327168000,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.59799999999999998},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.62},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.57099999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289692,"kb_used_data":1844,"kb_used_omap":0,"kb_used_meta":287807,"kb_avail":94082148,"statfs":{"total":96636764160,"available":96340119552,"internally_reserved":0,"allocated":1888256,"data_stored":1605051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":294714420},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":10.411},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":2.262}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.256}]}]}]}} 2021-01-15T18:51:14.625 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:14.951 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:14.953 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:14.964 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2021-01-15T18:51:14.880994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18596,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1192344,"kb_used_data":1784,"kb_used_omap":2,"kb_used_meta":1190525,"kb_avail":376295016,"statfs":{"total":386547056640,"available":385326096384,"internally_reserved":0,"allocated":1826816,"data_stored":693996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1219097756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2121752,"num_objects":-260,"num_object_clones":0,"num_object_copies":-1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.182186"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"9780","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:07.601021+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:07.601021+0000","last_peered":"2021-01-15T18:51:07.601021+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:07.601021+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12398","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.351225+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:07.351225+0000","last_peered":"2021-01-15T18:51:07.351225+0000","last_clean":"2021-01-15T18:51:07.351225+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:07.351225+0000","last_undegraded":"2021-01-15T18:51:07.351225+0000","last_fullsized":"2021-01-15T18:51:07.351225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12438","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.689288+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.689288+0000","last_peered":"2021-01-15T18:51:07.689288+0000","last_clean":"2021-01-15T18:51:07.689288+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.689288+0000","last_undegraded":"2021-01-15T18:51:07.689288+0000","last_fullsized":"2021-01-15T18:51:07.689288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10052","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:07.518917+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:07.518917+0000","last_peered":"2021-01-15T18:51:07.518917+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:07.518917+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10433","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.655788+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:07.655788+0000","last_peered":"2021-01-15T18:51:07.655788+0000","last_clean":"2021-01-15T18:51:07.655788+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:07.655788+0000","last_undegraded":"2021-01-15T18:51:07.655788+0000","last_fullsized":"2021-01-15T18:51:07.655788+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9750","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.597934+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:07.597934+0000","last_peered":"2021-01-15T18:51:07.597934+0000","last_clean":"2021-01-15T18:51:07.597934+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:07.597934+0000","last_undegraded":"2021-01-15T18:51:07.597934+0000","last_fullsized":"2021-01-15T18:51:07.597934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12599","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.436923+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:07.436923+0000","last_peered":"2021-01-15T18:51:07.436923+0000","last_clean":"2021-01-15T18:51:07.436923+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:07.436923+0000","last_undegraded":"2021-01-15T18:51:07.436923+0000","last_fullsized":"2021-01-15T18:51:07.436923+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12867","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.623695+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:07.623695+0000","last_peered":"2021-01-15T18:51:07.623695+0000","last_clean":"2021-01-15T18:51:07.623695+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:07.623695+0000","last_undegraded":"2021-01-15T18:51:07.623695+0000","last_fullsized":"2021-01-15T18:51:07.623695+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12677","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.640711+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:07.640711+0000","last_peered":"2021-01-15T18:51:07.640711+0000","last_clean":"2021-01-15T18:51:07.640711+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:07.640711+0000","last_undegraded":"2021-01-15T18:51:07.640711+0000","last_fullsized":"2021-01-15T18:51:07.640711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12711","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.566219+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:07.566219+0000","last_peered":"2021-01-15T18:51:07.566219+0000","last_clean":"2021-01-15T18:51:07.566219+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:07.566219+0000","last_undegraded":"2021-01-15T18:51:07.566219+0000","last_fullsized":"2021-01-15T18:51:07.566219+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12406","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.511695+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:07.511695+0000","last_peered":"2021-01-15T18:51:07.511695+0000","last_clean":"2021-01-15T18:51:07.511695+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:07.511695+0000","last_undegraded":"2021-01-15T18:51:07.511695+0000","last_fullsized":"2021-01-15T18:51:07.511695+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12409","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.347096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:07.347096+0000","last_peered":"2021-01-15T18:51:07.347096+0000","last_clean":"2021-01-15T18:51:07.347096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:07.347096+0000","last_undegraded":"2021-01-15T18:51:07.347096+0000","last_fullsized":"2021-01-15T18:51:07.347096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12383","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.487567+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:07.487567+0000","last_peered":"2021-01-15T18:51:07.487567+0000","last_clean":"2021-01-15T18:51:07.487567+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:07.487567+0000","last_undegraded":"2021-01-15T18:51:07.487567+0000","last_fullsized":"2021-01-15T18:51:07.487567+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10125","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.424950+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:07.424950+0000","last_peered":"2021-01-15T18:51:07.424950+0000","last_clean":"2021-01-15T18:51:07.424950+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:07.424950+0000","last_undegraded":"2021-01-15T18:51:07.424950+0000","last_fullsized":"2021-01-15T18:51:07.424950+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"12643","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:11.432201+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:11.432201+0000","last_peered":"2021-01-15T18:51:11.432201+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:11.432201+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1265,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12418","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.632013+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.632013+0000","last_peered":"2021-01-15T18:51:07.632013+0000","last_clean":"2021-01-15T18:51:07.632013+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.632013+0000","last_undegraded":"2021-01-15T18:51:07.632013+0000","last_fullsized":"2021-01-15T18:51:07.632013+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18596,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294084,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077756,"statfs":{"total":96636764160,"available":96335622144,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.52300000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.79600000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302340,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069500,"statfs":{"total":96636764160,"available":96327168000,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.59799999999999998},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.62},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.57099999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293452,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":292991,"kb_avail":94078388,"statfs":{"total":96636764160,"available":96336269312,"internally_reserved":0,"allocated":462848,"data_stored":179643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300022836},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.504},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.623}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.68300000000000005}]}]}]}} 2021-01-15T18:51:14.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:15.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:15.329 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2811/112 objects degraded (2509.821%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":725,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":1220960256,"bytes_avail":385326096384,"bytes_total":386547056640,"degraded_objects":2811,"degraded_total":112,"degraded_ratio":25.098214285714285,"recovering_objects_per_sec":45,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":453,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":102},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:15.343 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:15.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:15.663 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:15.678 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2021-01-15T18:51:14.880994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18596,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1192344,"kb_used_data":1784,"kb_used_omap":2,"kb_used_meta":1190525,"kb_avail":376295016,"statfs":{"total":386547056640,"available":385326096384,"internally_reserved":0,"allocated":1826816,"data_stored":693996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1219097756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2121752,"num_objects":-260,"num_object_clones":0,"num_object_copies":-1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.182186"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"9780","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:07.601021+0000","last_change":"2021-01-15T18:40:32.791596+0000","last_active":"2021-01-15T18:51:07.601021+0000","last_peered":"2021-01-15T18:51:07.601021+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:07.601021+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12398","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.351225+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:07.351225+0000","last_peered":"2021-01-15T18:51:07.351225+0000","last_clean":"2021-01-15T18:51:07.351225+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:07.351225+0000","last_undegraded":"2021-01-15T18:51:07.351225+0000","last_fullsized":"2021-01-15T18:51:07.351225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12438","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.689288+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.689288+0000","last_peered":"2021-01-15T18:51:07.689288+0000","last_clean":"2021-01-15T18:51:07.689288+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.689288+0000","last_undegraded":"2021-01-15T18:51:07.689288+0000","last_fullsized":"2021-01-15T18:51:07.689288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10052","reported_epoch":"54","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:07.518917+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:07.518917+0000","last_peered":"2021-01-15T18:51:07.518917+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:07.518917+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10433","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.655788+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:07.655788+0000","last_peered":"2021-01-15T18:51:07.655788+0000","last_clean":"2021-01-15T18:51:07.655788+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:07.655788+0000","last_undegraded":"2021-01-15T18:51:07.655788+0000","last_fullsized":"2021-01-15T18:51:07.655788+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9750","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.597934+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:07.597934+0000","last_peered":"2021-01-15T18:51:07.597934+0000","last_clean":"2021-01-15T18:51:07.597934+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:07.597934+0000","last_undegraded":"2021-01-15T18:51:07.597934+0000","last_fullsized":"2021-01-15T18:51:07.597934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12599","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.436923+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:07.436923+0000","last_peered":"2021-01-15T18:51:07.436923+0000","last_clean":"2021-01-15T18:51:07.436923+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:07.436923+0000","last_undegraded":"2021-01-15T18:51:07.436923+0000","last_fullsized":"2021-01-15T18:51:07.436923+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12867","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.623695+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:07.623695+0000","last_peered":"2021-01-15T18:51:07.623695+0000","last_clean":"2021-01-15T18:51:07.623695+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:07.623695+0000","last_undegraded":"2021-01-15T18:51:07.623695+0000","last_fullsized":"2021-01-15T18:51:07.623695+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12677","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.640711+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:07.640711+0000","last_peered":"2021-01-15T18:51:07.640711+0000","last_clean":"2021-01-15T18:51:07.640711+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:07.640711+0000","last_undegraded":"2021-01-15T18:51:07.640711+0000","last_fullsized":"2021-01-15T18:51:07.640711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12711","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.566219+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:07.566219+0000","last_peered":"2021-01-15T18:51:07.566219+0000","last_clean":"2021-01-15T18:51:07.566219+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:07.566219+0000","last_undegraded":"2021-01-15T18:51:07.566219+0000","last_fullsized":"2021-01-15T18:51:07.566219+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12406","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.511695+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:07.511695+0000","last_peered":"2021-01-15T18:51:07.511695+0000","last_clean":"2021-01-15T18:51:07.511695+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:07.511695+0000","last_undegraded":"2021-01-15T18:51:07.511695+0000","last_fullsized":"2021-01-15T18:51:07.511695+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12409","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.347096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:07.347096+0000","last_peered":"2021-01-15T18:51:07.347096+0000","last_clean":"2021-01-15T18:51:07.347096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:07.347096+0000","last_undegraded":"2021-01-15T18:51:07.347096+0000","last_fullsized":"2021-01-15T18:51:07.347096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12383","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.487567+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:07.487567+0000","last_peered":"2021-01-15T18:51:07.487567+0000","last_clean":"2021-01-15T18:51:07.487567+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:07.487567+0000","last_undegraded":"2021-01-15T18:51:07.487567+0000","last_fullsized":"2021-01-15T18:51:07.487567+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10125","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.424950+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:07.424950+0000","last_peered":"2021-01-15T18:51:07.424950+0000","last_clean":"2021-01-15T18:51:07.424950+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:07.424950+0000","last_undegraded":"2021-01-15T18:51:07.424950+0000","last_fullsized":"2021-01-15T18:51:07.424950+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"12643","reported_epoch":"54","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:11.432201+0000","last_change":"2021-01-15T18:49:48.689599+0000","last_active":"2021-01-15T18:51:11.432201+0000","last_peered":"2021-01-15T18:51:11.432201+0000","last_clean":"2021-01-15T18:39:34.420980+0000","last_became_active":"2021-01-15T18:40:32.762690+0000","last_became_peered":"2021-01-15T18:40:32.762690+0000","last_unstale":"2021-01-15T18:51:11.432201+0000","last_undegraded":"2021-01-15T18:40:32.529520+0000","last_fullsized":"2021-01-15T18:40:32.482105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1265,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-223}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12418","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.632013+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.632013+0000","last_peered":"2021-01-15T18:51:07.632013+0000","last_clean":"2021-01-15T18:51:07.632013+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.632013+0000","last_undegraded":"2021-01-15T18:51:07.632013+0000","last_fullsized":"2021-01-15T18:51:07.632013+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18596,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215244,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294084,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077756,"statfs":{"total":96636764160,"available":96335622144,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.52300000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.79600000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302340,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069500,"statfs":{"total":96636764160,"available":96327168000,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.59799999999999998},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.62},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.57099999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293452,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":292991,"kb_avail":94078388,"statfs":{"total":96636764160,"available":96336269312,"internally_reserved":0,"allocated":462848,"data_stored":179643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300022836},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.504},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.623}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.68300000000000005}]}]}]}} 2021-01-15T18:51:18.682 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:18.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:18.991 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:19.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2021-01-15T18:51:18.701182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19275,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1197376,"kb_used_data":1824,"kb_used_omap":2,"kb_used_meta":1195517,"kb_avail":376289984,"statfs":{"total":386547056640,"available":385320943616,"internally_reserved":0,"allocated":1867776,"data_stored":709574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1224209564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1384448,"num_objects":-169,"num_object_clones":0,"num_object_copies":-676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"10687","reported_epoch":"56","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:16.439761+0000","last_change":"2021-01-15T18:51:13.398749+0000","last_active":"2021-01-15T18:51:16.439761+0000","last_peered":"2021-01-15T18:51:16.439761+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:16.439761+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12400","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894450+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:13.894450+0000","last_peered":"2021-01-15T18:51:13.894450+0000","last_clean":"2021-01-15T18:51:13.894450+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:13.894450+0000","last_undegraded":"2021-01-15T18:51:13.894450+0000","last_fullsized":"2021-01-15T18:51:13.894450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12438","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.689288+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.689288+0000","last_peered":"2021-01-15T18:51:07.689288+0000","last_clean":"2021-01-15T18:51:07.689288+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.689288+0000","last_undegraded":"2021-01-15T18:51:07.689288+0000","last_fullsized":"2021-01-15T18:51:07.689288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10054","reported_epoch":"55","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:13.882185+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:13.882185+0000","last_peered":"2021-01-15T18:51:13.882185+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:13.882185+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10435","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894317+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:13.894317+0000","last_peered":"2021-01-15T18:51:13.894317+0000","last_clean":"2021-01-15T18:51:13.894317+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:13.894317+0000","last_undegraded":"2021-01-15T18:51:13.894317+0000","last_fullsized":"2021-01-15T18:51:13.894317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9752","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12601","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894206+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:13.894206+0000","last_peered":"2021-01-15T18:51:13.894206+0000","last_clean":"2021-01-15T18:51:13.894206+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:13.894206+0000","last_undegraded":"2021-01-15T18:51:13.894206+0000","last_fullsized":"2021-01-15T18:51:13.894206+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12869","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894124+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:13.894124+0000","last_peered":"2021-01-15T18:51:13.894124+0000","last_clean":"2021-01-15T18:51:13.894124+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:13.894124+0000","last_undegraded":"2021-01-15T18:51:13.894124+0000","last_fullsized":"2021-01-15T18:51:13.894124+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12679","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.898136+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:13.898136+0000","last_peered":"2021-01-15T18:51:13.898136+0000","last_clean":"2021-01-15T18:51:13.898136+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:13.898136+0000","last_undegraded":"2021-01-15T18:51:13.898136+0000","last_fullsized":"2021-01-15T18:51:13.898136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12713","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881147+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:13.881147+0000","last_peered":"2021-01-15T18:51:13.881147+0000","last_clean":"2021-01-15T18:51:13.881147+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:13.881147+0000","last_undegraded":"2021-01-15T18:51:13.881147+0000","last_fullsized":"2021-01-15T18:51:13.881147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12408","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.880780+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:13.880780+0000","last_peered":"2021-01-15T18:51:13.880780+0000","last_clean":"2021-01-15T18:51:13.880780+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:13.880780+0000","last_undegraded":"2021-01-15T18:51:13.880780+0000","last_fullsized":"2021-01-15T18:51:13.880780+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12411","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:13.894096+0000","last_peered":"2021-01-15T18:51:13.894096+0000","last_clean":"2021-01-15T18:51:13.894096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:13.894096+0000","last_undegraded":"2021-01-15T18:51:13.894096+0000","last_fullsized":"2021-01-15T18:51:13.894096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12385","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881674+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:13.881674+0000","last_peered":"2021-01-15T18:51:13.881674+0000","last_clean":"2021-01-15T18:51:13.881674+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:13.881674+0000","last_undegraded":"2021-01-15T18:51:13.881674+0000","last_fullsized":"2021-01-15T18:51:13.881674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10127","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13110","reported_epoch":"56","state":"active+clean","last_fresh":"2021-01-15T18:51:14.986662+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:14.986662+0000","last_peered":"2021-01-15T18:51:14.986662+0000","last_clean":"2021-01-15T18:51:14.986662+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:14.986662+0000","last_undegraded":"2021-01-15T18:51:14.986662+0000","last_fullsized":"2021-01-15T18:51:14.986662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12418","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.632013+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.632013+0000","last_peered":"2021-01-15T18:51:07.632013+0000","last_clean":"2021-01-15T18:51:07.632013+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.632013+0000","last_undegraded":"2021-01-15T18:51:07.632013+0000","last_fullsized":"2021-01-15T18:51:07.632013+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19275,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294100,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077740,"statfs":{"total":96636764160,"available":96335605760,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.73799999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.63}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.79600000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302348,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069492,"statfs":{"total":96636764160,"available":96327159808,"internally_reserved":0,"allocated":462848,"data_stored":174575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.38700000000000001},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.59899999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298460,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":297983,"kb_avail":94073380,"statfs":{"total":96636764160,"available":96331141120,"internally_reserved":0,"allocated":479232,"data_stored":185870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":305134644},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.38},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.751}]}]}]}} 2021-01-15T18:51:19.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:19.368 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:19.369 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2132/112 objects degraded (1903.571%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":729,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":56,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":1226113024,"bytes_avail":385320943616,"bytes_total":386547056640,"degraded_objects":2132,"degraded_total":112,"degraded_ratio":19.035714285714285,"recovering_objects_per_sec":135,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":21},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:19.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:19.715 INFO:teuthology.orchestra.run.smithi148.stderr:2021-01-15T18:51:19.714+0000 7faa827fc700 0 --2- 172.21.15.148:0/4169530277 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7faa8411aff0 0x7faa84120da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-01-15T18:51:19.912 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:19.914 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:19.927 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2021-01-15T18:51:18.701182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19275,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1197376,"kb_used_data":1824,"kb_used_omap":2,"kb_used_meta":1195517,"kb_avail":376289984,"statfs":{"total":386547056640,"available":385320943616,"internally_reserved":0,"allocated":1867776,"data_stored":709574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1224209564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1384448,"num_objects":-169,"num_object_clones":0,"num_object_copies":-676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"10687","reported_epoch":"56","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:16.439761+0000","last_change":"2021-01-15T18:51:13.398749+0000","last_active":"2021-01-15T18:51:16.439761+0000","last_peered":"2021-01-15T18:51:16.439761+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:16.439761+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12400","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894450+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:13.894450+0000","last_peered":"2021-01-15T18:51:13.894450+0000","last_clean":"2021-01-15T18:51:13.894450+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:13.894450+0000","last_undegraded":"2021-01-15T18:51:13.894450+0000","last_fullsized":"2021-01-15T18:51:13.894450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12438","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.689288+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:07.689288+0000","last_peered":"2021-01-15T18:51:07.689288+0000","last_clean":"2021-01-15T18:51:07.689288+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:07.689288+0000","last_undegraded":"2021-01-15T18:51:07.689288+0000","last_fullsized":"2021-01-15T18:51:07.689288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10054","reported_epoch":"55","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:13.882185+0000","last_change":"2021-01-15T18:40:32.811946+0000","last_active":"2021-01-15T18:51:13.882185+0000","last_peered":"2021-01-15T18:51:13.882185+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:13.882185+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10435","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894317+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:13.894317+0000","last_peered":"2021-01-15T18:51:13.894317+0000","last_clean":"2021-01-15T18:51:13.894317+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:13.894317+0000","last_undegraded":"2021-01-15T18:51:13.894317+0000","last_fullsized":"2021-01-15T18:51:13.894317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9752","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12601","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894206+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:13.894206+0000","last_peered":"2021-01-15T18:51:13.894206+0000","last_clean":"2021-01-15T18:51:13.894206+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:13.894206+0000","last_undegraded":"2021-01-15T18:51:13.894206+0000","last_fullsized":"2021-01-15T18:51:13.894206+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12869","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894124+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:13.894124+0000","last_peered":"2021-01-15T18:51:13.894124+0000","last_clean":"2021-01-15T18:51:13.894124+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:13.894124+0000","last_undegraded":"2021-01-15T18:51:13.894124+0000","last_fullsized":"2021-01-15T18:51:13.894124+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12679","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.898136+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:13.898136+0000","last_peered":"2021-01-15T18:51:13.898136+0000","last_clean":"2021-01-15T18:51:13.898136+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:13.898136+0000","last_undegraded":"2021-01-15T18:51:13.898136+0000","last_fullsized":"2021-01-15T18:51:13.898136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12713","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881147+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:13.881147+0000","last_peered":"2021-01-15T18:51:13.881147+0000","last_clean":"2021-01-15T18:51:13.881147+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:13.881147+0000","last_undegraded":"2021-01-15T18:51:13.881147+0000","last_fullsized":"2021-01-15T18:51:13.881147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12408","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.880780+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:13.880780+0000","last_peered":"2021-01-15T18:51:13.880780+0000","last_clean":"2021-01-15T18:51:13.880780+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:13.880780+0000","last_undegraded":"2021-01-15T18:51:13.880780+0000","last_fullsized":"2021-01-15T18:51:13.880780+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12411","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:13.894096+0000","last_peered":"2021-01-15T18:51:13.894096+0000","last_clean":"2021-01-15T18:51:13.894096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:13.894096+0000","last_undegraded":"2021-01-15T18:51:13.894096+0000","last_fullsized":"2021-01-15T18:51:13.894096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12385","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881674+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:13.881674+0000","last_peered":"2021-01-15T18:51:13.881674+0000","last_clean":"2021-01-15T18:51:13.881674+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:13.881674+0000","last_undegraded":"2021-01-15T18:51:13.881674+0000","last_fullsized":"2021-01-15T18:51:13.881674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10127","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13110","reported_epoch":"56","state":"active+clean","last_fresh":"2021-01-15T18:51:14.986662+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:14.986662+0000","last_peered":"2021-01-15T18:51:14.986662+0000","last_clean":"2021-01-15T18:51:14.986662+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:14.986662+0000","last_undegraded":"2021-01-15T18:51:14.986662+0000","last_fullsized":"2021-01-15T18:51:14.986662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12418","reported_epoch":"54","state":"active+clean","last_fresh":"2021-01-15T18:51:07.632013+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:07.632013+0000","last_peered":"2021-01-15T18:51:07.632013+0000","last_clean":"2021-01-15T18:51:07.632013+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:07.632013+0000","last_undegraded":"2021-01-15T18:51:07.632013+0000","last_fullsized":"2021-01-15T18:51:07.632013+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19275,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215245,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294100,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077740,"statfs":{"total":96636764160,"available":96335605760,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":6.149,"5min":3.5419999999999998,"15min":3.4159999999999999},"min":{"1min":0.61499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":38.780999999999999,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.085,"5min":2.4500000000000002,"15min":3.157},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":42.018999999999998,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.73799999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:16 2021","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":4.5110000000000001,"15min":3.2490000000000001},"min":{"1min":0.50600000000000001,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":25.425000000000001,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":4.3659999999999997,"5min":4.3010000000000002,"15min":3.7629999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":25.233000000000001,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.63}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":454656,"data_stored":171451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.79600000000000004}]}]},{"osd":0,"up_from":22,"seq":94489280658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302348,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":301887,"kb_avail":94069492,"statfs":{"total":96636764160,"available":96327159808,"internally_reserved":0,"allocated":462848,"data_stored":174575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309132340},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.38700000000000001},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.59899999999999998}]}]},{"osd":1,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298460,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":297983,"kb_avail":94073380,"statfs":{"total":96636764160,"available":96331141120,"internally_reserved":0,"allocated":479232,"data_stored":185870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":305134644},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.38},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.751}]}]}]}} 2021-01-15T18:51:22.929 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:23.254 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:23.256 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:23.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2021-01-15T18:51:22.914875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20118,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1198616,"kb_used_data":1848,"kb_used_omap":2,"kb_used_meta":1196733,"kb_avail":376288744,"statfs":{"total":386547056640,"available":385319673856,"internally_reserved":0,"allocated":1892352,"data_stored":718904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1225454748},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1522,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.033881"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12284","reported_epoch":"56","state":"active+undersized+remapped","last_fresh":"2021-01-15T18:51:20.948027+0000","last_change":"2021-01-15T18:51:20.948350+0000","last_active":"2021-01-15T18:51:20.948027+0000","last_peered":"2021-01-15T18:51:20.948027+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:20.948027+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12400","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894450+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:13.894450+0000","last_peered":"2021-01-15T18:51:13.894450+0000","last_clean":"2021-01-15T18:51:13.894450+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:13.894450+0000","last_undegraded":"2021-01-15T18:51:13.894450+0000","last_fullsized":"2021-01-15T18:51:13.894450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12439","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.883196+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:13.883196+0000","last_peered":"2021-01-15T18:51:13.883196+0000","last_clean":"2021-01-15T18:51:13.883196+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:13.883196+0000","last_undegraded":"2021-01-15T18:51:13.883196+0000","last_fullsized":"2021-01-15T18:51:13.883196+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10145","reported_epoch":"56","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:21.437578+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:21.437578+0000","last_peered":"2021-01-15T18:51:21.437578+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:21.437578+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10435","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894317+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:13.894317+0000","last_peered":"2021-01-15T18:51:13.894317+0000","last_clean":"2021-01-15T18:51:13.894317+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:13.894317+0000","last_undegraded":"2021-01-15T18:51:13.894317+0000","last_fullsized":"2021-01-15T18:51:13.894317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9752","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12601","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894206+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:13.894206+0000","last_peered":"2021-01-15T18:51:13.894206+0000","last_clean":"2021-01-15T18:51:13.894206+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:13.894206+0000","last_undegraded":"2021-01-15T18:51:13.894206+0000","last_fullsized":"2021-01-15T18:51:13.894206+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12869","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894124+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:13.894124+0000","last_peered":"2021-01-15T18:51:13.894124+0000","last_clean":"2021-01-15T18:51:13.894124+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:13.894124+0000","last_undegraded":"2021-01-15T18:51:13.894124+0000","last_fullsized":"2021-01-15T18:51:13.894124+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12679","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.898136+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:13.898136+0000","last_peered":"2021-01-15T18:51:13.898136+0000","last_clean":"2021-01-15T18:51:13.898136+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:13.898136+0000","last_undegraded":"2021-01-15T18:51:13.898136+0000","last_fullsized":"2021-01-15T18:51:13.898136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12713","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881147+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:13.881147+0000","last_peered":"2021-01-15T18:51:13.881147+0000","last_clean":"2021-01-15T18:51:13.881147+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:13.881147+0000","last_undegraded":"2021-01-15T18:51:13.881147+0000","last_fullsized":"2021-01-15T18:51:13.881147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12408","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.880780+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:13.880780+0000","last_peered":"2021-01-15T18:51:13.880780+0000","last_clean":"2021-01-15T18:51:13.880780+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:13.880780+0000","last_undegraded":"2021-01-15T18:51:13.880780+0000","last_fullsized":"2021-01-15T18:51:13.880780+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12411","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:13.894096+0000","last_peered":"2021-01-15T18:51:13.894096+0000","last_clean":"2021-01-15T18:51:13.894096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:13.894096+0000","last_undegraded":"2021-01-15T18:51:13.894096+0000","last_fullsized":"2021-01-15T18:51:13.894096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12385","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881674+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:13.881674+0000","last_peered":"2021-01-15T18:51:13.881674+0000","last_clean":"2021-01-15T18:51:13.881674+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:13.881674+0000","last_undegraded":"2021-01-15T18:51:13.881674+0000","last_fullsized":"2021-01-15T18:51:13.881674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10127","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13110","reported_epoch":"56","state":"active+clean","last_fresh":"2021-01-15T18:51:14.986662+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:14.986662+0000","last_peered":"2021-01-15T18:51:14.986662+0000","last_clean":"2021-01-15T18:51:14.986662+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:14.986662+0000","last_undegraded":"2021-01-15T18:51:14.986662+0000","last_fullsized":"2021-01-15T18:51:14.986662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12419","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.883307+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:13.883307+0000","last_peered":"2021-01-15T18:51:13.883307+0000","last_clean":"2021-01-15T18:51:13.883307+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:13.883307+0000","last_undegraded":"2021-01-15T18:51:13.883307+0000","last_fullsized":"2021-01-15T18:51:13.883307+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20118,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294100,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077740,"statfs":{"total":96636764160,"available":96335605760,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.78400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302484,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069356,"statfs":{"total":96636764160,"available":96327020544,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.60899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.876},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.73999999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302420,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069420,"statfs":{"total":96636764160,"available":96327086080,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42199999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.80200000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299612,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":299135,"kb_avail":94072228,"statfs":{"total":96636764160,"available":96329961472,"internally_reserved":0,"allocated":479232,"data_stored":185870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":306314292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.67100000000000004}]}]}]}} 2021-01-15T18:51:23.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:23.627 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:23.739 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1289/112 objects degraded (1150.893%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":734,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":1227382784,"bytes_avail":385319673856,"bytes_total":386547056640,"degraded_objects":1289,"degraded_total":112,"degraded_ratio":11.508928571428571,"recovering_objects_per_sec":190,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1522,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:23.740 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:23.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:24.071 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:24.101 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:24.101 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2021-01-15T18:51:22.914875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20118,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1198616,"kb_used_data":1848,"kb_used_omap":2,"kb_used_meta":1196733,"kb_avail":376288744,"statfs":{"total":386547056640,"available":385319673856,"internally_reserved":0,"allocated":1892352,"data_stored":718904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1225454748},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1522,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.033881"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12284","reported_epoch":"56","state":"active+undersized+remapped","last_fresh":"2021-01-15T18:51:20.948027+0000","last_change":"2021-01-15T18:51:20.948350+0000","last_active":"2021-01-15T18:51:20.948027+0000","last_peered":"2021-01-15T18:51:20.948027+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.763152+0000","last_became_peered":"2021-01-15T18:40:32.763152+0000","last_unstale":"2021-01-15T18:51:20.948027+0000","last_undegraded":"2021-01-15T18:40:32.522130+0000","last_fullsized":"2021-01-15T18:40:32.484560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","1(2)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12400","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894450+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:13.894450+0000","last_peered":"2021-01-15T18:51:13.894450+0000","last_clean":"2021-01-15T18:51:13.894450+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:13.894450+0000","last_undegraded":"2021-01-15T18:51:13.894450+0000","last_fullsized":"2021-01-15T18:51:13.894450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12439","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.883196+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:13.883196+0000","last_peered":"2021-01-15T18:51:13.883196+0000","last_clean":"2021-01-15T18:51:13.883196+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:13.883196+0000","last_undegraded":"2021-01-15T18:51:13.883196+0000","last_fullsized":"2021-01-15T18:51:13.883196+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"10145","reported_epoch":"56","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:21.437578+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:21.437578+0000","last_peered":"2021-01-15T18:51:21.437578+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:21.437578+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10435","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894317+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:13.894317+0000","last_peered":"2021-01-15T18:51:13.894317+0000","last_clean":"2021-01-15T18:51:13.894317+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:13.894317+0000","last_undegraded":"2021-01-15T18:51:13.894317+0000","last_fullsized":"2021-01-15T18:51:13.894317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9752","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12601","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894206+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:13.894206+0000","last_peered":"2021-01-15T18:51:13.894206+0000","last_clean":"2021-01-15T18:51:13.894206+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:13.894206+0000","last_undegraded":"2021-01-15T18:51:13.894206+0000","last_fullsized":"2021-01-15T18:51:13.894206+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12869","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894124+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:13.894124+0000","last_peered":"2021-01-15T18:51:13.894124+0000","last_clean":"2021-01-15T18:51:13.894124+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:13.894124+0000","last_undegraded":"2021-01-15T18:51:13.894124+0000","last_fullsized":"2021-01-15T18:51:13.894124+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12679","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.898136+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:13.898136+0000","last_peered":"2021-01-15T18:51:13.898136+0000","last_clean":"2021-01-15T18:51:13.898136+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:13.898136+0000","last_undegraded":"2021-01-15T18:51:13.898136+0000","last_fullsized":"2021-01-15T18:51:13.898136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12713","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881147+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:13.881147+0000","last_peered":"2021-01-15T18:51:13.881147+0000","last_clean":"2021-01-15T18:51:13.881147+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:13.881147+0000","last_undegraded":"2021-01-15T18:51:13.881147+0000","last_fullsized":"2021-01-15T18:51:13.881147+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12408","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.880780+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:13.880780+0000","last_peered":"2021-01-15T18:51:13.880780+0000","last_clean":"2021-01-15T18:51:13.880780+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:13.880780+0000","last_undegraded":"2021-01-15T18:51:13.880780+0000","last_fullsized":"2021-01-15T18:51:13.880780+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12411","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894096+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:13.894096+0000","last_peered":"2021-01-15T18:51:13.894096+0000","last_clean":"2021-01-15T18:51:13.894096+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:13.894096+0000","last_undegraded":"2021-01-15T18:51:13.894096+0000","last_fullsized":"2021-01-15T18:51:13.894096+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12385","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.881674+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:13.881674+0000","last_peered":"2021-01-15T18:51:13.881674+0000","last_clean":"2021-01-15T18:51:13.881674+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:13.881674+0000","last_undegraded":"2021-01-15T18:51:13.881674+0000","last_fullsized":"2021-01-15T18:51:13.881674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10127","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.894220+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:13.894220+0000","last_peered":"2021-01-15T18:51:13.894220+0000","last_clean":"2021-01-15T18:51:13.894220+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:13.894220+0000","last_undegraded":"2021-01-15T18:51:13.894220+0000","last_fullsized":"2021-01-15T18:51:13.894220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13110","reported_epoch":"56","state":"active+clean","last_fresh":"2021-01-15T18:51:14.986662+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:14.986662+0000","last_peered":"2021-01-15T18:51:14.986662+0000","last_clean":"2021-01-15T18:51:14.986662+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:14.986662+0000","last_undegraded":"2021-01-15T18:51:14.986662+0000","last_fullsized":"2021-01-15T18:51:14.986662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12419","reported_epoch":"55","state":"active+clean","last_fresh":"2021-01-15T18:51:13.883307+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:13.883307+0000","last_peered":"2021-01-15T18:51:13.883307+0000","last_clean":"2021-01-15T18:51:13.883307+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:13.883307+0000","last_undegraded":"2021-01-15T18:51:13.883307+0000","last_fullsized":"2021-01-15T18:51:13.883307+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20118,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215246,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294100,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":293631,"kb_avail":94077740,"statfs":{"total":96636764160,"available":96335605760,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300678196},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.78400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302484,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069356,"statfs":{"total":96636764160,"available":96327020544,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.60899999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.876},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.73999999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302420,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069420,"statfs":{"total":96636764160,"available":96327086080,"internally_reserved":0,"allocated":471040,"data_stored":177678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42199999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.80200000000000005}]}]},{"osd":1,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299612,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":299135,"kb_avail":94072228,"statfs":{"total":96636764160,"available":96329961472,"internally_reserved":0,"allocated":479232,"data_stored":185870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":306314292},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.67100000000000004}]}]}]}} 2021-01-15T18:51:27.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:27.427 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:27.428 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:27.440 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2021-01-15T18:51:26.702508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191512,"kb_used_data":1912,"kb_used_omap":2,"kb_used_meta":1189565,"kb_avail":376295848,"statfs":{"total":386547056640,"available":385326948352,"internally_reserved":0,"allocated":1957888,"data_stored":743516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1218114716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12296","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:23.017021+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:23.017021+0000","last_peered":"2021-01-15T18:51:23.017021+0000","last_clean":"2021-01-15T18:51:23.017021+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:23.017021+0000","last_undegraded":"2021-01-15T18:51:23.017021+0000","last_fullsized":"2021-01-15T18:51:23.017021+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12402","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920562+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:21.920562+0000","last_peered":"2021-01-15T18:51:21.920562+0000","last_clean":"2021-01-15T18:51:21.920562+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:21.920562+0000","last_undegraded":"2021-01-15T18:51:21.920562+0000","last_fullsized":"2021-01-15T18:51:21.920562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12441","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.928942+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:21.928942+0000","last_peered":"2021-01-15T18:51:21.928942+0000","last_clean":"2021-01-15T18:51:21.928942+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:21.928942+0000","last_undegraded":"2021-01-15T18:51:21.928942+0000","last_fullsized":"2021-01-15T18:51:21.928942+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"11721","reported_epoch":"58","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:26.440716+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:26.440716+0000","last_peered":"2021-01-15T18:51:26.440716+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:26.440716+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10437","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920305+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:21.920305+0000","last_peered":"2021-01-15T18:51:21.920305+0000","last_clean":"2021-01-15T18:51:21.920305+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:21.920305+0000","last_undegraded":"2021-01-15T18:51:21.920305+0000","last_fullsized":"2021-01-15T18:51:21.920305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9754","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:21.920053+0000","last_peered":"2021-01-15T18:51:21.920053+0000","last_clean":"2021-01-15T18:51:21.920053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:21.920053+0000","last_undegraded":"2021-01-15T18:51:21.920053+0000","last_fullsized":"2021-01-15T18:51:21.920053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12603","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919816+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:21.919816+0000","last_peered":"2021-01-15T18:51:21.919816+0000","last_clean":"2021-01-15T18:51:21.919816+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:21.919816+0000","last_undegraded":"2021-01-15T18:51:21.919816+0000","last_fullsized":"2021-01-15T18:51:21.919816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12871","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919671+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:21.919671+0000","last_peered":"2021-01-15T18:51:21.919671+0000","last_clean":"2021-01-15T18:51:21.919671+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:21.919671+0000","last_undegraded":"2021-01-15T18:51:21.919671+0000","last_fullsized":"2021-01-15T18:51:21.919671+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12681","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.922890+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:21.922890+0000","last_peered":"2021-01-15T18:51:21.922890+0000","last_clean":"2021-01-15T18:51:21.922890+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:21.922890+0000","last_undegraded":"2021-01-15T18:51:21.922890+0000","last_fullsized":"2021-01-15T18:51:21.922890+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12715","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915842+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:21.915842+0000","last_peered":"2021-01-15T18:51:21.915842+0000","last_clean":"2021-01-15T18:51:21.915842+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:21.915842+0000","last_undegraded":"2021-01-15T18:51:21.915842+0000","last_fullsized":"2021-01-15T18:51:21.915842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12410","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915571+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:21.915571+0000","last_peered":"2021-01-15T18:51:21.915571+0000","last_clean":"2021-01-15T18:51:21.915571+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:21.915571+0000","last_undegraded":"2021-01-15T18:51:21.915571+0000","last_fullsized":"2021-01-15T18:51:21.915571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12413","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919638+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:21.919638+0000","last_peered":"2021-01-15T18:51:21.919638+0000","last_clean":"2021-01-15T18:51:21.919638+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:21.919638+0000","last_undegraded":"2021-01-15T18:51:21.919638+0000","last_fullsized":"2021-01-15T18:51:21.919638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12387","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.926648+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:21.926648+0000","last_peered":"2021-01-15T18:51:21.926648+0000","last_clean":"2021-01-15T18:51:21.926648+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:21.926648+0000","last_undegraded":"2021-01-15T18:51:21.926648+0000","last_fullsized":"2021-01-15T18:51:21.926648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10129","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920172+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:21.920172+0000","last_peered":"2021-01-15T18:51:21.920172+0000","last_clean":"2021-01-15T18:51:21.920172+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:21.920172+0000","last_undegraded":"2021-01-15T18:51:21.920172+0000","last_fullsized":"2021-01-15T18:51:21.920172+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13112","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915776+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:21.915776+0000","last_peered":"2021-01-15T18:51:21.915776+0000","last_clean":"2021-01-15T18:51:21.915776+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:21.915776+0000","last_undegraded":"2021-01-15T18:51:21.915776+0000","last_fullsized":"2021-01-15T18:51:21.915776+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12421","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.929016+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:21.929016+0000","last_peered":"2021-01-15T18:51:21.929016+0000","last_clean":"2021-01-15T18:51:21.929016+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:21.929016+0000","last_undegraded":"2021-01-15T18:51:21.929016+0000","last_fullsized":"2021-01-15T18:51:21.929016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294180,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077660,"statfs":{"total":96636764160,"available":96335523840,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.622},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302500,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069340,"statfs":{"total":96636764160,"available":96327004160,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.85199999999999998},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.40899999999999997},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.79900000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292396,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":291903,"kb_avail":94079444,"statfs":{"total":96636764160,"available":96337350656,"internally_reserved":0,"allocated":495616,"data_stored":192023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298908724},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.5},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89500000000000002}]}]}]}} 2021-01-15T18:51:27.443 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:27.799 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:27.799 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 502/112 objects degraded (448.214%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":1220108288,"bytes_avail":385326948352,"bytes_total":386547056640,"degraded_objects":502,"degraded_total":112,"degraded_ratio":4.4821428571428568,"recovering_objects_per_sec":203,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1630,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:27.813 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:27.813 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:28.114 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:28.116 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:28.127 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2021-01-15T18:51:26.702508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191512,"kb_used_data":1912,"kb_used_omap":2,"kb_used_meta":1189565,"kb_avail":376295848,"statfs":{"total":386547056640,"available":385326948352,"internally_reserved":0,"allocated":1957888,"data_stored":743516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1218114716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1630,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12296","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:23.017021+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:23.017021+0000","last_peered":"2021-01-15T18:51:23.017021+0000","last_clean":"2021-01-15T18:51:23.017021+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:23.017021+0000","last_undegraded":"2021-01-15T18:51:23.017021+0000","last_fullsized":"2021-01-15T18:51:23.017021+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12402","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920562+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:21.920562+0000","last_peered":"2021-01-15T18:51:21.920562+0000","last_clean":"2021-01-15T18:51:21.920562+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:21.920562+0000","last_undegraded":"2021-01-15T18:51:21.920562+0000","last_fullsized":"2021-01-15T18:51:21.920562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12441","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.928942+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:21.928942+0000","last_peered":"2021-01-15T18:51:21.928942+0000","last_clean":"2021-01-15T18:51:21.928942+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:21.928942+0000","last_undegraded":"2021-01-15T18:51:21.928942+0000","last_fullsized":"2021-01-15T18:51:21.928942+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"11721","reported_epoch":"58","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:26.440716+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:26.440716+0000","last_peered":"2021-01-15T18:51:26.440716+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:26.440716+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10437","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920305+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:21.920305+0000","last_peered":"2021-01-15T18:51:21.920305+0000","last_clean":"2021-01-15T18:51:21.920305+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:21.920305+0000","last_undegraded":"2021-01-15T18:51:21.920305+0000","last_fullsized":"2021-01-15T18:51:21.920305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9754","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:21.920053+0000","last_peered":"2021-01-15T18:51:21.920053+0000","last_clean":"2021-01-15T18:51:21.920053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:21.920053+0000","last_undegraded":"2021-01-15T18:51:21.920053+0000","last_fullsized":"2021-01-15T18:51:21.920053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12603","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919816+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:21.919816+0000","last_peered":"2021-01-15T18:51:21.919816+0000","last_clean":"2021-01-15T18:51:21.919816+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:21.919816+0000","last_undegraded":"2021-01-15T18:51:21.919816+0000","last_fullsized":"2021-01-15T18:51:21.919816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12871","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919671+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:21.919671+0000","last_peered":"2021-01-15T18:51:21.919671+0000","last_clean":"2021-01-15T18:51:21.919671+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:21.919671+0000","last_undegraded":"2021-01-15T18:51:21.919671+0000","last_fullsized":"2021-01-15T18:51:21.919671+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12681","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.922890+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:21.922890+0000","last_peered":"2021-01-15T18:51:21.922890+0000","last_clean":"2021-01-15T18:51:21.922890+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:21.922890+0000","last_undegraded":"2021-01-15T18:51:21.922890+0000","last_fullsized":"2021-01-15T18:51:21.922890+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12715","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915842+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:21.915842+0000","last_peered":"2021-01-15T18:51:21.915842+0000","last_clean":"2021-01-15T18:51:21.915842+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:21.915842+0000","last_undegraded":"2021-01-15T18:51:21.915842+0000","last_fullsized":"2021-01-15T18:51:21.915842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12410","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915571+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:21.915571+0000","last_peered":"2021-01-15T18:51:21.915571+0000","last_clean":"2021-01-15T18:51:21.915571+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:21.915571+0000","last_undegraded":"2021-01-15T18:51:21.915571+0000","last_fullsized":"2021-01-15T18:51:21.915571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12413","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919638+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:21.919638+0000","last_peered":"2021-01-15T18:51:21.919638+0000","last_clean":"2021-01-15T18:51:21.919638+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:21.919638+0000","last_undegraded":"2021-01-15T18:51:21.919638+0000","last_fullsized":"2021-01-15T18:51:21.919638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12387","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.926648+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:21.926648+0000","last_peered":"2021-01-15T18:51:21.926648+0000","last_clean":"2021-01-15T18:51:21.926648+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:21.926648+0000","last_undegraded":"2021-01-15T18:51:21.926648+0000","last_fullsized":"2021-01-15T18:51:21.926648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10129","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920172+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:21.920172+0000","last_peered":"2021-01-15T18:51:21.920172+0000","last_clean":"2021-01-15T18:51:21.920172+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:21.920172+0000","last_undegraded":"2021-01-15T18:51:21.920172+0000","last_fullsized":"2021-01-15T18:51:21.920172+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13112","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915776+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:21.915776+0000","last_peered":"2021-01-15T18:51:21.915776+0000","last_clean":"2021-01-15T18:51:21.915776+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:21.915776+0000","last_undegraded":"2021-01-15T18:51:21.915776+0000","last_fullsized":"2021-01-15T18:51:21.915776+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12421","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.929016+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:21.929016+0000","last_peered":"2021-01-15T18:51:21.929016+0000","last_clean":"2021-01-15T18:51:21.929016+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:21.929016+0000","last_undegraded":"2021-01-15T18:51:21.929016+0000","last_fullsized":"2021-01-15T18:51:21.929016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294180,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077660,"statfs":{"total":96636764160,"available":96335523840,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.622},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302500,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069340,"statfs":{"total":96636764160,"available":96327004160,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.85199999999999998},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":22,"seq":94489280660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.40899999999999997},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.79900000000000004}]}]},{"osd":1,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292396,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":291903,"kb_avail":94079444,"statfs":{"total":96636764160,"available":96337350656,"internally_reserved":0,"allocated":495616,"data_stored":192023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298908724},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.5},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89500000000000002}]}]}]}} 2021-01-15T18:51:31.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:31.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:31.403 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:31.414 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2021-01-15T18:51:30.782840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191512,"kb_used_data":1912,"kb_used_omap":2,"kb_used_meta":1189565,"kb_avail":376295848,"statfs":{"total":386547056640,"available":385326948352,"internally_reserved":0,"allocated":1957888,"data_stored":743516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1218114716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.867964"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12296","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:23.017021+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:23.017021+0000","last_peered":"2021-01-15T18:51:23.017021+0000","last_clean":"2021-01-15T18:51:23.017021+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:23.017021+0000","last_undegraded":"2021-01-15T18:51:23.017021+0000","last_fullsized":"2021-01-15T18:51:23.017021+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12402","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920562+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:21.920562+0000","last_peered":"2021-01-15T18:51:21.920562+0000","last_clean":"2021-01-15T18:51:21.920562+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:21.920562+0000","last_undegraded":"2021-01-15T18:51:21.920562+0000","last_fullsized":"2021-01-15T18:51:21.920562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12441","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.928942+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:21.928942+0000","last_peered":"2021-01-15T18:51:21.928942+0000","last_clean":"2021-01-15T18:51:21.928942+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:21.928942+0000","last_undegraded":"2021-01-15T18:51:21.928942+0000","last_fullsized":"2021-01-15T18:51:21.928942+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"11721","reported_epoch":"58","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:26.440716+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:26.440716+0000","last_peered":"2021-01-15T18:51:26.440716+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:26.440716+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10437","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920305+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:21.920305+0000","last_peered":"2021-01-15T18:51:21.920305+0000","last_clean":"2021-01-15T18:51:21.920305+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:21.920305+0000","last_undegraded":"2021-01-15T18:51:21.920305+0000","last_fullsized":"2021-01-15T18:51:21.920305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9754","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:21.920053+0000","last_peered":"2021-01-15T18:51:21.920053+0000","last_clean":"2021-01-15T18:51:21.920053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:21.920053+0000","last_undegraded":"2021-01-15T18:51:21.920053+0000","last_fullsized":"2021-01-15T18:51:21.920053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12603","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919816+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:21.919816+0000","last_peered":"2021-01-15T18:51:21.919816+0000","last_clean":"2021-01-15T18:51:21.919816+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:21.919816+0000","last_undegraded":"2021-01-15T18:51:21.919816+0000","last_fullsized":"2021-01-15T18:51:21.919816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12871","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919671+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:21.919671+0000","last_peered":"2021-01-15T18:51:21.919671+0000","last_clean":"2021-01-15T18:51:21.919671+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:21.919671+0000","last_undegraded":"2021-01-15T18:51:21.919671+0000","last_fullsized":"2021-01-15T18:51:21.919671+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12681","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.922890+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:21.922890+0000","last_peered":"2021-01-15T18:51:21.922890+0000","last_clean":"2021-01-15T18:51:21.922890+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:21.922890+0000","last_undegraded":"2021-01-15T18:51:21.922890+0000","last_fullsized":"2021-01-15T18:51:21.922890+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12715","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915842+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:21.915842+0000","last_peered":"2021-01-15T18:51:21.915842+0000","last_clean":"2021-01-15T18:51:21.915842+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:21.915842+0000","last_undegraded":"2021-01-15T18:51:21.915842+0000","last_fullsized":"2021-01-15T18:51:21.915842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12410","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915571+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:21.915571+0000","last_peered":"2021-01-15T18:51:21.915571+0000","last_clean":"2021-01-15T18:51:21.915571+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:21.915571+0000","last_undegraded":"2021-01-15T18:51:21.915571+0000","last_fullsized":"2021-01-15T18:51:21.915571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12413","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919638+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:21.919638+0000","last_peered":"2021-01-15T18:51:21.919638+0000","last_clean":"2021-01-15T18:51:21.919638+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:21.919638+0000","last_undegraded":"2021-01-15T18:51:21.919638+0000","last_fullsized":"2021-01-15T18:51:21.919638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12388","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:22.921224+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:22.921224+0000","last_peered":"2021-01-15T18:51:22.921224+0000","last_clean":"2021-01-15T18:51:22.921224+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:22.921224+0000","last_undegraded":"2021-01-15T18:51:22.921224+0000","last_fullsized":"2021-01-15T18:51:22.921224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10129","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920172+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:21.920172+0000","last_peered":"2021-01-15T18:51:21.920172+0000","last_clean":"2021-01-15T18:51:21.920172+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:21.920172+0000","last_undegraded":"2021-01-15T18:51:21.920172+0000","last_fullsized":"2021-01-15T18:51:21.920172+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13112","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915776+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:21.915776+0000","last_peered":"2021-01-15T18:51:21.915776+0000","last_clean":"2021-01-15T18:51:21.915776+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:21.915776+0000","last_undegraded":"2021-01-15T18:51:21.915776+0000","last_fullsized":"2021-01-15T18:51:21.915776+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12421","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.929016+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:21.929016+0000","last_peered":"2021-01-15T18:51:21.929016+0000","last_clean":"2021-01-15T18:51:21.929016+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:21.929016+0000","last_undegraded":"2021-01-15T18:51:21.929016+0000","last_fullsized":"2021-01-15T18:51:21.929016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294180,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077660,"statfs":{"total":96636764160,"available":96335523840,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.622},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302500,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069340,"statfs":{"total":96636764160,"available":96327004160,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.66000000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.751},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.65700000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292396,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":291903,"kb_avail":94079444,"statfs":{"total":96636764160,"available":96337350656,"internally_reserved":0,"allocated":495616,"data_stored":192023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298908724},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.5},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89500000000000002}]}]}]}} 2021-01-15T18:51:31.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-01-15T18:51:31.732 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:31.733 INFO:teuthology.orchestra.run.smithi148.stdout:{"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 502/112 objects degraded (448.214%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":742,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":60,"num_osds":4,"num_up_osds":4,"osd_up_since":1610736030,"num_in_osds":4,"osd_in_since":1610735953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":117012,"bytes_used":1220108288,"bytes_avail":385326948352,"bytes_total":386547056640,"degraded_objects":502,"degraded_total":112,"degraded_ratio":4.4821428571428568,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2021-01-15T18:50:40.679970+0000","services":{}},"progress_events":{}} 2021-01-15T18:51:31.745 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2021-01-15T18:51:31.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:32.015 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:32.016 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:32.027 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2021-01-15T18:51:30.782840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191512,"kb_used_data":1912,"kb_used_omap":2,"kb_used_meta":1189565,"kb_avail":376295848,"statfs":{"total":386547056640,"available":385326948352,"internally_reserved":0,"allocated":1957888,"data_stored":743516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1218114716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.867964"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12296","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:23.017021+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:23.017021+0000","last_peered":"2021-01-15T18:51:23.017021+0000","last_clean":"2021-01-15T18:51:23.017021+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:23.017021+0000","last_undegraded":"2021-01-15T18:51:23.017021+0000","last_fullsized":"2021-01-15T18:51:23.017021+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12402","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920562+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:21.920562+0000","last_peered":"2021-01-15T18:51:21.920562+0000","last_clean":"2021-01-15T18:51:21.920562+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:21.920562+0000","last_undegraded":"2021-01-15T18:51:21.920562+0000","last_fullsized":"2021-01-15T18:51:21.920562+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12441","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.928942+0000","last_change":"2021-01-15T18:46:30.754513+0000","last_active":"2021-01-15T18:51:21.928942+0000","last_peered":"2021-01-15T18:51:21.928942+0000","last_clean":"2021-01-15T18:51:21.928942+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:21.928942+0000","last_undegraded":"2021-01-15T18:51:21.928942+0000","last_fullsized":"2021-01-15T18:51:21.928942+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9491,"num_write_kb":37955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"11721","reported_epoch":"58","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-01-15T18:51:26.440716+0000","last_change":"2021-01-15T18:51:20.949794+0000","last_active":"2021-01-15T18:51:26.440716+0000","last_peered":"2021-01-15T18:51:26.440716+0000","last_clean":"2021-01-15T18:39:25.483201+0000","last_became_active":"2021-01-15T18:40:32.799577+0000","last_became_peered":"2021-01-15T18:40:32.799577+0000","last_unstale":"2021-01-15T18:51:26.440716+0000","last_undegraded":"2021-01-15T18:40:32.543550+0000","last_fullsized":"2021-01-15T18:40:32.532847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10437","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920305+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:21.920305+0000","last_peered":"2021-01-15T18:51:21.920305+0000","last_clean":"2021-01-15T18:51:21.920305+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:21.920305+0000","last_undegraded":"2021-01-15T18:51:21.920305+0000","last_fullsized":"2021-01-15T18:51:21.920305+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9754","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920053+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:21.920053+0000","last_peered":"2021-01-15T18:51:21.920053+0000","last_clean":"2021-01-15T18:51:21.920053+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:21.920053+0000","last_undegraded":"2021-01-15T18:51:21.920053+0000","last_fullsized":"2021-01-15T18:51:21.920053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12603","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919816+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:21.919816+0000","last_peered":"2021-01-15T18:51:21.919816+0000","last_clean":"2021-01-15T18:51:21.919816+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:21.919816+0000","last_undegraded":"2021-01-15T18:51:21.919816+0000","last_fullsized":"2021-01-15T18:51:21.919816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12871","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919671+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:21.919671+0000","last_peered":"2021-01-15T18:51:21.919671+0000","last_clean":"2021-01-15T18:51:21.919671+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:21.919671+0000","last_undegraded":"2021-01-15T18:51:21.919671+0000","last_fullsized":"2021-01-15T18:51:21.919671+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12681","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.922890+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:21.922890+0000","last_peered":"2021-01-15T18:51:21.922890+0000","last_clean":"2021-01-15T18:51:21.922890+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:21.922890+0000","last_undegraded":"2021-01-15T18:51:21.922890+0000","last_fullsized":"2021-01-15T18:51:21.922890+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12715","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915842+0000","last_change":"2021-01-15T18:42:01.754276+0000","last_active":"2021-01-15T18:51:21.915842+0000","last_peered":"2021-01-15T18:51:21.915842+0000","last_clean":"2021-01-15T18:51:21.915842+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:21.915842+0000","last_undegraded":"2021-01-15T18:51:21.915842+0000","last_fullsized":"2021-01-15T18:51:21.915842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12537,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9753,"num_write_kb":39012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":11714560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12410","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915571+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:21.915571+0000","last_peered":"2021-01-15T18:51:21.915571+0000","last_clean":"2021-01-15T18:51:21.915571+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:21.915571+0000","last_undegraded":"2021-01-15T18:51:21.915571+0000","last_fullsized":"2021-01-15T18:51:21.915571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12413","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.919638+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:21.919638+0000","last_peered":"2021-01-15T18:51:21.919638+0000","last_clean":"2021-01-15T18:51:21.919638+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:21.919638+0000","last_undegraded":"2021-01-15T18:51:21.919638+0000","last_fullsized":"2021-01-15T18:51:21.919638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12388","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:22.921224+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:22.921224+0000","last_peered":"2021-01-15T18:51:22.921224+0000","last_clean":"2021-01-15T18:51:22.921224+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:22.921224+0000","last_undegraded":"2021-01-15T18:51:22.921224+0000","last_fullsized":"2021-01-15T18:51:22.921224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10129","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.920172+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:21.920172+0000","last_peered":"2021-01-15T18:51:21.920172+0000","last_clean":"2021-01-15T18:51:21.920172+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:21.920172+0000","last_undegraded":"2021-01-15T18:51:21.920172+0000","last_fullsized":"2021-01-15T18:51:21.920172+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13112","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.915776+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:21.915776+0000","last_peered":"2021-01-15T18:51:21.915776+0000","last_clean":"2021-01-15T18:51:21.915776+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:21.915776+0000","last_undegraded":"2021-01-15T18:51:21.915776+0000","last_fullsized":"2021-01-15T18:51:21.915776+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12421","reported_epoch":"57","state":"active+clean","last_fresh":"2021-01-15T18:51:21.929016+0000","last_change":"2021-01-15T18:44:24.732139+0000","last_active":"2021-01-15T18:51:21.929016+0000","last_peered":"2021-01-15T18:51:21.929016+0000","last_clean":"2021-01-15T18:51:21.929016+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:21.929016+0000","last_undegraded":"2021-01-15T18:51:21.929016+0000","last_fullsized":"2021-01-15T18:51:21.929016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":38146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11321344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":117012,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154697,"num_write_kb":618791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20905,"num_bytes_recovered":68804608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294180,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077660,"statfs":{"total":96636764160,"available":96335523840,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.622},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.83299999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302500,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":302016,"kb_avail":94069340,"statfs":{"total":96636764160,"available":96327004160,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309264384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.51700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.66000000000000003}]}]},{"osd":0,"up_from":22,"seq":94489280661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.751},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.65700000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":292396,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":291903,"kb_avail":94079444,"statfs":{"total":96636764160,"available":96337350656,"internally_reserved":0,"allocated":495616,"data_stored":192023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":298908724},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.5},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.89500000000000002}]}]}]}} 2021-01-15T18:51:35.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:35.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:35.304 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:35.315 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2021-01-15T18:51:34.704387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91938,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125917,"num_write_kb":503678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17167,"num_bytes_recovered":45768704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1192448,"kb_used_data":1952,"kb_used_omap":2,"kb_used_meta":1190461,"kb_avail":376294912,"statfs":{"total":386547056640,"available":385325989888,"internally_reserved":0,"allocated":1998848,"data_stored":758724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1219032220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25074,"num_objects":-6,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-28780,"num_write_kb":-115113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3738,"num_bytes_recovered":-23035904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001879"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12298","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788848+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:29.788848+0000","last_peered":"2021-01-15T18:51:29.788848+0000","last_clean":"2021-01-15T18:51:29.788848+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:29.788848+0000","last_undegraded":"2021-01-15T18:51:29.788848+0000","last_fullsized":"2021-01-15T18:51:29.788848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12404","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796727+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:29.796727+0000","last_peered":"2021-01-15T18:51:29.796727+0000","last_clean":"2021-01-15T18:51:29.796727+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:29.796727+0000","last_undegraded":"2021-01-15T18:51:29.796727+0000","last_fullsized":"2021-01-15T18:51:29.796727+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12447","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.828565+0000","last_change":"2021-01-15T18:51:30.828677+0000","last_active":"2021-01-15T18:51:30.828565+0000","last_peered":"2021-01-15T18:51:30.828565+0000","last_clean":"2021-01-15T18:51:30.828565+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:30.828565+0000","last_undegraded":"2021-01-15T18:51:30.828565+0000","last_fullsized":"2021-01-15T18:51:30.828565+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9492","last_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_deep_scrub":"54'9492","last_deep_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_clean_scrub_stamp":"2021-01-15T18:51:30.828559+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"12738","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.868511+0000","last_change":"2021-01-15T18:51:30.868511+0000","last_active":"2021-01-15T18:51:30.868511+0000","last_peered":"2021-01-15T18:51:30.868511+0000","last_clean":"2021-01-15T18:51:30.868511+0000","last_became_active":"2021-01-15T18:51:30.803006+0000","last_became_peered":"2021-01-15T18:51:30.803006+0000","last_unstale":"2021-01-15T18:51:30.868511+0000","last_undegraded":"2021-01-15T18:51:30.868511+0000","last_fullsized":"2021-01-15T18:51:30.868511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10439","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796611+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:29.796611+0000","last_peered":"2021-01-15T18:51:29.796611+0000","last_clean":"2021-01-15T18:51:29.796611+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:29.796611+0000","last_undegraded":"2021-01-15T18:51:29.796611+0000","last_fullsized":"2021-01-15T18:51:29.796611+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9756","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796708+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:29.796708+0000","last_peered":"2021-01-15T18:51:29.796708+0000","last_clean":"2021-01-15T18:51:29.796708+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:29.796708+0000","last_undegraded":"2021-01-15T18:51:29.796708+0000","last_fullsized":"2021-01-15T18:51:29.796708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12605","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796628+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:29.796628+0000","last_peered":"2021-01-15T18:51:29.796628+0000","last_clean":"2021-01-15T18:51:29.796628+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:29.796628+0000","last_undegraded":"2021-01-15T18:51:29.796628+0000","last_fullsized":"2021-01-15T18:51:29.796628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12873","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796403+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:29.796403+0000","last_peered":"2021-01-15T18:51:29.796403+0000","last_clean":"2021-01-15T18:51:29.796403+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:29.796403+0000","last_undegraded":"2021-01-15T18:51:29.796403+0000","last_fullsized":"2021-01-15T18:51:29.796403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12683","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796673+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:29.796673+0000","last_peered":"2021-01-15T18:51:29.796673+0000","last_clean":"2021-01-15T18:51:29.796673+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:29.796673+0000","last_undegraded":"2021-01-15T18:51:29.796673+0000","last_fullsized":"2021-01-15T18:51:29.796673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12721","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788780+0000","last_change":"2021-01-15T18:51:29.594840+0000","last_active":"2021-01-15T18:51:29.788780+0000","last_peered":"2021-01-15T18:51:29.788780+0000","last_clean":"2021-01-15T18:51:29.788780+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:29.788780+0000","last_undegraded":"2021-01-15T18:51:29.788780+0000","last_fullsized":"2021-01-15T18:51:29.788780+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9756","last_scrub_stamp":"2021-01-15T18:51:29.594666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.594666+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12412","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788213+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:29.788213+0000","last_peered":"2021-01-15T18:51:29.788213+0000","last_clean":"2021-01-15T18:51:29.788213+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:29.788213+0000","last_undegraded":"2021-01-15T18:51:29.788213+0000","last_fullsized":"2021-01-15T18:51:29.788213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12415","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796400+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:29.796400+0000","last_peered":"2021-01-15T18:51:29.796400+0000","last_clean":"2021-01-15T18:51:29.796400+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:29.796400+0000","last_undegraded":"2021-01-15T18:51:29.796400+0000","last_fullsized":"2021-01-15T18:51:29.796400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12388","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:22.921224+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:22.921224+0000","last_peered":"2021-01-15T18:51:22.921224+0000","last_clean":"2021-01-15T18:51:22.921224+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:22.921224+0000","last_undegraded":"2021-01-15T18:51:22.921224+0000","last_fullsized":"2021-01-15T18:51:22.921224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10131","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796559+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:29.796559+0000","last_peered":"2021-01-15T18:51:29.796559+0000","last_clean":"2021-01-15T18:51:29.796559+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:29.796559+0000","last_undegraded":"2021-01-15T18:51:29.796559+0000","last_fullsized":"2021-01-15T18:51:29.796559+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13114","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788299+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:29.788299+0000","last_peered":"2021-01-15T18:51:29.788299+0000","last_clean":"2021-01-15T18:51:29.788299+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:29.788299+0000","last_undegraded":"2021-01-15T18:51:29.788299+0000","last_fullsized":"2021-01-15T18:51:29.788299+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12427","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.799754+0000","last_change":"2021-01-15T18:51:29.775922+0000","last_active":"2021-01-15T18:51:29.799754+0000","last_peered":"2021-01-15T18:51:29.799754+0000","last_clean":"2021-01-15T18:51:29.799754+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:29.799754+0000","last_undegraded":"2021-01-15T18:51:29.799754+0000","last_fullsized":"2021-01-15T18:51:29.799754+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9538","last_scrub_stamp":"2021-01-15T18:51:29.775771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.775771+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91938,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125917,"num_write_kb":503678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17167,"num_bytes_recovered":45768704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294188,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077652,"statfs":{"total":96636764160,"available":96335515648,"internally_reserved":0,"allocated":495616,"data_stored":186881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.84799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302580,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302080,"kb_avail":94069260,"statfs":{"total":96636764160,"available":96326922240,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309329920},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.65100000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.82699999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.751},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.65700000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293244,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":292735,"kb_avail":94078596,"statfs":{"total":96636764160,"available":96336482304,"internally_reserved":0,"allocated":512000,"data_stored":198102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":299760692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.0760000000000001},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.85299999999999998}]}]}]}} 2021-01-15T18:51:35.316 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2021-01-15T18:51:35.317 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-01-15T18:51:35.326 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2021-01-15T18:51:35.326 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:51:35.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:35.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":60,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:51:30.779120+0000","last_up_change":"2021-01-15T18:40:30.471863+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2021-01-15T18:39:24.508766+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":44,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":38392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":38392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":38392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":38392}]},"public_addr":"172.21.15.148:6804/38392","cluster_addr":"172.21.15.148:6805/38392","heartbeat_back_addr":"172.21.15.148:6807/38392","heartbeat_front_addr":"172.21.15.148:6806/38392","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":26,"up_thru":51,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":43629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":43629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":43629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":43629}]},"public_addr":"172.21.15.148:6812/43629","cluster_addr":"172.21.15.148:6813/43629","heartbeat_back_addr":"172.21.15.148:6815/43629","heartbeat_front_addr":"172.21.15.148:6814/43629","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":24,"up_thru":59,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":39252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":39252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":39252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":39252}]},"public_addr":"172.21.15.148:6808/39252","cluster_addr":"172.21.15.148:6809/39252","heartbeat_back_addr":"172.21.15.148:6811/39252","heartbeat_front_addr":"172.21.15.148:6810/39252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:51:35.599 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:35.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:35.866 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2021-01-15T18:51:34.704387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91938,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125917,"num_write_kb":503678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17167,"num_bytes_recovered":45768704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1192448,"kb_used_data":1952,"kb_used_omap":2,"kb_used_meta":1190461,"kb_avail":376294912,"statfs":{"total":386547056640,"available":385325989888,"internally_reserved":0,"allocated":1998848,"data_stored":758724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1219032220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25074,"num_objects":-6,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-28780,"num_write_kb":-115113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3738,"num_bytes_recovered":-23035904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001879"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12298","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788848+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:29.788848+0000","last_peered":"2021-01-15T18:51:29.788848+0000","last_clean":"2021-01-15T18:51:29.788848+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:29.788848+0000","last_undegraded":"2021-01-15T18:51:29.788848+0000","last_fullsized":"2021-01-15T18:51:29.788848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12404","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796727+0000","last_change":"2021-01-15T18:41:36.654440+0000","last_active":"2021-01-15T18:51:29.796727+0000","last_peered":"2021-01-15T18:51:29.796727+0000","last_clean":"2021-01-15T18:51:29.796727+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:29.796727+0000","last_undegraded":"2021-01-15T18:51:29.796727+0000","last_fullsized":"2021-01-15T18:51:29.796727+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1422,"num_bytes_recovered":11616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12447","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.828565+0000","last_change":"2021-01-15T18:51:30.828677+0000","last_active":"2021-01-15T18:51:30.828565+0000","last_peered":"2021-01-15T18:51:30.828565+0000","last_clean":"2021-01-15T18:51:30.828565+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:30.828565+0000","last_undegraded":"2021-01-15T18:51:30.828565+0000","last_fullsized":"2021-01-15T18:51:30.828565+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9492","last_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_deep_scrub":"54'9492","last_deep_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_clean_scrub_stamp":"2021-01-15T18:51:30.828559+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"12738","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.868511+0000","last_change":"2021-01-15T18:51:30.868511+0000","last_active":"2021-01-15T18:51:30.868511+0000","last_peered":"2021-01-15T18:51:30.868511+0000","last_clean":"2021-01-15T18:51:30.868511+0000","last_became_active":"2021-01-15T18:51:30.803006+0000","last_became_peered":"2021-01-15T18:51:30.803006+0000","last_unstale":"2021-01-15T18:51:30.868511+0000","last_undegraded":"2021-01-15T18:51:30.868511+0000","last_fullsized":"2021-01-15T18:51:30.868511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10439","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796611+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:29.796611+0000","last_peered":"2021-01-15T18:51:29.796611+0000","last_clean":"2021-01-15T18:51:29.796611+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:29.796611+0000","last_undegraded":"2021-01-15T18:51:29.796611+0000","last_fullsized":"2021-01-15T18:51:29.796611+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9756","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796708+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:29.796708+0000","last_peered":"2021-01-15T18:51:29.796708+0000","last_clean":"2021-01-15T18:51:29.796708+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:29.796708+0000","last_undegraded":"2021-01-15T18:51:29.796708+0000","last_fullsized":"2021-01-15T18:51:29.796708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12605","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796628+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:29.796628+0000","last_peered":"2021-01-15T18:51:29.796628+0000","last_clean":"2021-01-15T18:51:29.796628+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:29.796628+0000","last_undegraded":"2021-01-15T18:51:29.796628+0000","last_fullsized":"2021-01-15T18:51:29.796628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12873","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796403+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:29.796403+0000","last_peered":"2021-01-15T18:51:29.796403+0000","last_clean":"2021-01-15T18:51:29.796403+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:29.796403+0000","last_undegraded":"2021-01-15T18:51:29.796403+0000","last_fullsized":"2021-01-15T18:51:29.796403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12683","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796673+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:29.796673+0000","last_peered":"2021-01-15T18:51:29.796673+0000","last_clean":"2021-01-15T18:51:29.796673+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:29.796673+0000","last_undegraded":"2021-01-15T18:51:29.796673+0000","last_fullsized":"2021-01-15T18:51:29.796673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12721","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788780+0000","last_change":"2021-01-15T18:51:29.594840+0000","last_active":"2021-01-15T18:51:29.788780+0000","last_peered":"2021-01-15T18:51:29.788780+0000","last_clean":"2021-01-15T18:51:29.788780+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:29.788780+0000","last_undegraded":"2021-01-15T18:51:29.788780+0000","last_fullsized":"2021-01-15T18:51:29.788780+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9756","last_scrub_stamp":"2021-01-15T18:51:29.594666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.594666+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12412","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788213+0000","last_change":"2021-01-15T18:49:50.269728+0000","last_active":"2021-01-15T18:51:29.788213+0000","last_peered":"2021-01-15T18:51:29.788213+0000","last_clean":"2021-01-15T18:51:29.788213+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:29.788213+0000","last_undegraded":"2021-01-15T18:51:29.788213+0000","last_fullsized":"2021-01-15T18:51:29.788213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":38207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12415","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796400+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:29.796400+0000","last_peered":"2021-01-15T18:51:29.796400+0000","last_clean":"2021-01-15T18:51:29.796400+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:29.796400+0000","last_undegraded":"2021-01-15T18:51:29.796400+0000","last_fullsized":"2021-01-15T18:51:29.796400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12388","reported_epoch":"58","state":"active+clean","last_fresh":"2021-01-15T18:51:22.921224+0000","last_change":"2021-01-15T18:43:09.671044+0000","last_active":"2021-01-15T18:51:22.921224+0000","last_peered":"2021-01-15T18:51:22.921224+0000","last_clean":"2021-01-15T18:51:22.921224+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:22.921224+0000","last_undegraded":"2021-01-15T18:51:22.921224+0000","last_fullsized":"2021-01-15T18:51:22.921224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":37884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":11509760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10131","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796559+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:29.796559+0000","last_peered":"2021-01-15T18:51:29.796559+0000","last_clean":"2021-01-15T18:51:29.796559+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:29.796559+0000","last_undegraded":"2021-01-15T18:51:29.796559+0000","last_fullsized":"2021-01-15T18:51:29.796559+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13114","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788299+0000","last_change":"2021-01-15T18:51:14.986662+0000","last_active":"2021-01-15T18:51:29.788299+0000","last_peered":"2021-01-15T18:51:29.788299+0000","last_clean":"2021-01-15T18:51:29.788299+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:29.788299+0000","last_undegraded":"2021-01-15T18:51:29.788299+0000","last_fullsized":"2021-01-15T18:51:29.788299+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16716,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":40147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12427","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.799754+0000","last_change":"2021-01-15T18:51:29.775922+0000","last_active":"2021-01-15T18:51:29.799754+0000","last_peered":"2021-01-15T18:51:29.799754+0000","last_clean":"2021-01-15T18:51:29.799754+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:29.799754+0000","last_undegraded":"2021-01-15T18:51:29.799754+0000","last_fullsized":"2021-01-15T18:51:29.799754+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9538","last_scrub_stamp":"2021-01-15T18:51:29.775771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.775771+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":91938,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125917,"num_write_kb":503678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17167,"num_bytes_recovered":45768704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215248,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294188,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077652,"statfs":{"total":96636764160,"available":96335515648,"internally_reserved":0,"allocated":495616,"data_stored":186881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.84799999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302580,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302080,"kb_avail":94069260,"statfs":{"total":96636764160,"available":96326922240,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309329920},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.65100000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.82699999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":487424,"data_stored":183831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.81999999999999995}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.751},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.65700000000000003}]}]},{"osd":1,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293244,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":292735,"kb_avail":94078596,"statfs":{"total":96636764160,"available":96336482304,"internally_reserved":0,"allocated":512000,"data_stored":198102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":299760692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":1.0760000000000001},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.85299999999999998}]}]}]}} 2021-01-15T18:51:35.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-01-15T18:51:36.128 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:36.128 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":60,"fsid":"327970d8-d19c-43f6-b60e-6ccadab35a2c","created":"2021-01-15T18:39:04.487862+0000","modified":"2021-01-15T18:51:30.779120+0000","last_up_change":"2021-01-15T18:40:30.471863+0000","last_in_change":"2021-01-15T18:39:13.090654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2021-01-15T18:39:24.508766+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"107303da-44d7-4a60-915e-f6d1f55fa7cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":44,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":38392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6805","nonce":38392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":38392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":38392}]},"public_addr":"172.21.15.148:6804/38392","cluster_addr":"172.21.15.148:6805/38392","heartbeat_back_addr":"172.21.15.148:6807/38392","heartbeat_front_addr":"172.21.15.148:6806/38392","state":["exists","up"]},{"osd":1,"uuid":"d68efcd0-3df2-4b26-aa97-e09b936838a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":26,"up_thru":51,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":43629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":43629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":43629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":43629}]},"public_addr":"172.21.15.148:6812/43629","cluster_addr":"172.21.15.148:6813/43629","heartbeat_back_addr":"172.21.15.148:6815/43629","heartbeat_front_addr":"172.21.15.148:6814/43629","state":["exists","up"]},{"osd":2,"uuid":"755bf833-e4ce-4574-b5d3-90babe41a2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":31655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6801","nonce":31655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":31655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":31655}]},"public_addr":"172.21.15.148:6800/31655","cluster_addr":"172.21.15.148:6801/31655","heartbeat_back_addr":"172.21.15.148:6803/31655","heartbeat_front_addr":"172.21.15.148:6802/31655","state":["exists","up"]},{"osd":3,"uuid":"c524c862-86a3-4d88-b92c-ecb219ba36f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":24,"up_thru":59,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":39252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":39252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6811","nonce":39252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":39252}]},"public_addr":"172.21.15.148:6808/39252","cluster_addr":"172.21.15.148:6809/39252","heartbeat_back_addr":"172.21.15.148:6811/39252","heartbeat_front_addr":"172.21.15.148:6810/39252","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.890609+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.868335+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.834449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T18:39:10.872754+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-01-15T18:51:37.141 INFO:tasks.ceph:Scrubbing osd.0 2021-01-15T18:51:37.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2021-01-15T18:51:37.286 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:51:37.287 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-01-15T18:51:37.287 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:51:37.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2021-01-15T18:51:37.561 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 0 to deep-scrub 2021-01-15T18:51:37.572 INFO:tasks.ceph:Scrubbing osd.1 2021-01-15T18:51:37.572 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2021-01-15T18:51:37.703 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:51:37.704 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-01-15T18:51:37.704 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:51:37.714 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2021-01-15T18:51:37.977 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 1 to deep-scrub 2021-01-15T18:51:37.988 INFO:tasks.ceph:Scrubbing osd.2 2021-01-15T18:51:37.989 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2021-01-15T18:51:38.121 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:51:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-01-15T18:51:38.121 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:51:38.132 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-01-15T18:51:38.391 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 2 to deep-scrub 2021-01-15T18:51:38.401 INFO:tasks.ceph:Scrubbing osd.3 2021-01-15T18:51:38.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2021-01-15T18:51:38.533 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2021-01-15T18:51:38.534 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-01-15T18:51:38.534 INFO:teuthology.orchestra.run.smithi148.stdout:} 2021-01-15T18:51:38.545 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2021-01-15T18:51:38.805 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 3 to deep-scrub 2021-01-15T18:51:38.815 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:39.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:39.078 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:39.089 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2021-01-15T18:51:38.705016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54327,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87399,"num_write_kb":349604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11459,"num_bytes_recovered":22642688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1192472,"kb_used_data":1976,"kb_used_omap":2,"kb_used_meta":1190461,"kb_avail":376294888,"statfs":{"total":386547056640,"available":385325965312,"internally_reserved":0,"allocated":2023424,"data_stored":767832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1219032220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62685,"num_objects":-15,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-67298,"num_write_kb":-269187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9446,"num_bytes_recovered":-46161920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.926957"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12298","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788848+0000","last_change":"2021-01-15T18:51:23.017021+0000","last_active":"2021-01-15T18:51:29.788848+0000","last_peered":"2021-01-15T18:51:29.788848+0000","last_clean":"2021-01-15T18:51:29.788848+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:29.788848+0000","last_undegraded":"2021-01-15T18:51:29.788848+0000","last_fullsized":"2021-01-15T18:51:29.788848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12408","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:31.879513+0000","last_change":"2021-01-15T18:51:31.879612+0000","last_active":"2021-01-15T18:51:31.879513+0000","last_peered":"2021-01-15T18:51:31.879513+0000","last_clean":"2021-01-15T18:51:31.879513+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:31.879513+0000","last_undegraded":"2021-01-15T18:51:31.879513+0000","last_fullsized":"2021-01-15T18:51:31.879513+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9460","last_scrub_stamp":"2021-01-15T18:51:31.879512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:31.879512+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12447","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.828565+0000","last_change":"2021-01-15T18:51:30.828677+0000","last_active":"2021-01-15T18:51:30.828565+0000","last_peered":"2021-01-15T18:51:30.828565+0000","last_clean":"2021-01-15T18:51:30.828565+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:30.828565+0000","last_undegraded":"2021-01-15T18:51:30.828565+0000","last_fullsized":"2021-01-15T18:51:30.828565+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9492","last_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_deep_scrub":"54'9492","last_deep_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_clean_scrub_stamp":"2021-01-15T18:51:30.828559+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"12738","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.868511+0000","last_change":"2021-01-15T18:51:30.868511+0000","last_active":"2021-01-15T18:51:30.868511+0000","last_peered":"2021-01-15T18:51:30.868511+0000","last_clean":"2021-01-15T18:51:30.868511+0000","last_became_active":"2021-01-15T18:51:30.803006+0000","last_became_peered":"2021-01-15T18:51:30.803006+0000","last_unstale":"2021-01-15T18:51:30.868511+0000","last_undegraded":"2021-01-15T18:51:30.868511+0000","last_fullsized":"2021-01-15T18:51:30.868511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10439","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796611+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:29.796611+0000","last_peered":"2021-01-15T18:51:29.796611+0000","last_clean":"2021-01-15T18:51:29.796611+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:29.796611+0000","last_undegraded":"2021-01-15T18:51:29.796611+0000","last_fullsized":"2021-01-15T18:51:29.796611+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9756","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796708+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:29.796708+0000","last_peered":"2021-01-15T18:51:29.796708+0000","last_clean":"2021-01-15T18:51:29.796708+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:29.796708+0000","last_undegraded":"2021-01-15T18:51:29.796708+0000","last_fullsized":"2021-01-15T18:51:29.796708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12605","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796628+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:29.796628+0000","last_peered":"2021-01-15T18:51:29.796628+0000","last_clean":"2021-01-15T18:51:29.796628+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:29.796628+0000","last_undegraded":"2021-01-15T18:51:29.796628+0000","last_fullsized":"2021-01-15T18:51:29.796628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12873","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796403+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:29.796403+0000","last_peered":"2021-01-15T18:51:29.796403+0000","last_clean":"2021-01-15T18:51:29.796403+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:29.796403+0000","last_undegraded":"2021-01-15T18:51:29.796403+0000","last_fullsized":"2021-01-15T18:51:29.796403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12683","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796673+0000","last_change":"2021-01-15T18:46:09.009648+0000","last_active":"2021-01-15T18:51:29.796673+0000","last_peered":"2021-01-15T18:51:29.796673+0000","last_clean":"2021-01-15T18:51:29.796673+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:29.796673+0000","last_undegraded":"2021-01-15T18:51:29.796673+0000","last_fullsized":"2021-01-15T18:51:29.796673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9727,"num_write_kb":38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12721","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.788780+0000","last_change":"2021-01-15T18:51:29.594840+0000","last_active":"2021-01-15T18:51:29.788780+0000","last_peered":"2021-01-15T18:51:29.788780+0000","last_clean":"2021-01-15T18:51:29.788780+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:29.788780+0000","last_undegraded":"2021-01-15T18:51:29.788780+0000","last_fullsized":"2021-01-15T18:51:29.788780+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9756","last_scrub_stamp":"2021-01-15T18:51:29.594666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.594666+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12416","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:31.556904+0000","last_change":"2021-01-15T18:51:31.557007+0000","last_active":"2021-01-15T18:51:31.556904+0000","last_peered":"2021-01-15T18:51:31.556904+0000","last_clean":"2021-01-15T18:51:31.556904+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:31.556904+0000","last_undegraded":"2021-01-15T18:51:31.556904+0000","last_fullsized":"2021-01-15T18:51:31.556904+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9553","last_scrub_stamp":"2021-01-15T18:51:31.556904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:31.556904+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12415","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796400+0000","last_change":"2021-01-15T18:45:44.861720+0000","last_active":"2021-01-15T18:51:29.796400+0000","last_peered":"2021-01-15T18:51:29.796400+0000","last_clean":"2021-01-15T18:51:29.796400+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:29.796400+0000","last_undegraded":"2021-01-15T18:51:29.796400+0000","last_fullsized":"2021-01-15T18:51:29.796400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9564,"num_write_kb":38255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":8716288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12393","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:30.626267+0000","last_change":"2021-01-15T18:51:30.626267+0000","last_active":"2021-01-15T18:51:30.626267+0000","last_peered":"2021-01-15T18:51:30.626267+0000","last_clean":"2021-01-15T18:51:30.626267+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:30.626267+0000","last_undegraded":"2021-01-15T18:51:30.626267+0000","last_fullsized":"2021-01-15T18:51:30.626267+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9474","last_scrub_stamp":"2021-01-15T18:51:30.626149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:30.626149+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10131","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796559+0000","last_change":"2021-01-15T18:46:17.015479+0000","last_active":"2021-01-15T18:51:29.796559+0000","last_peered":"2021-01-15T18:51:29.796559+0000","last_clean":"2021-01-15T18:51:29.796559+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:29.796559+0000","last_undegraded":"2021-01-15T18:51:29.796559+0000","last_fullsized":"2021-01-15T18:51:29.796559+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":39532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13118","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:32.601325+0000","last_change":"2021-01-15T18:51:32.602320+0000","last_active":"2021-01-15T18:51:32.601325+0000","last_peered":"2021-01-15T18:51:32.601325+0000","last_clean":"2021-01-15T18:51:32.601325+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:32.601325+0000","last_undegraded":"2021-01-15T18:51:32.601325+0000","last_fullsized":"2021-01-15T18:51:32.601325+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'10040","last_scrub_stamp":"2021-01-15T18:51:32.601324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:32.601324+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12427","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.799754+0000","last_change":"2021-01-15T18:51:29.775922+0000","last_active":"2021-01-15T18:51:29.799754+0000","last_peered":"2021-01-15T18:51:29.799754+0000","last_clean":"2021-01-15T18:51:29.799754+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:29.799754+0000","last_undegraded":"2021-01-15T18:51:29.799754+0000","last_fullsized":"2021-01-15T18:51:29.799754+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9538","last_scrub_stamp":"2021-01-15T18:51:29.775771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:29.775771+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54327,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87399,"num_write_kb":349604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11459,"num_bytes_recovered":22642688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215249,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294196,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":293695,"kb_avail":94077644,"statfs":{"total":96636764160,"available":96335507456,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":300743732},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Fri Jan 15 18:50:45 2021","interfaces":[{"interface":"back","average":{"1min":3.7989999999999999,"5min":4.54,"15min":4.0469999999999997},"min":{"1min":0.57299999999999995,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":17.565000000000001,"5min":27.218,"15min":30.129000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":4.3330000000000002,"5min":4.9249999999999998,"15min":3.6659999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":12.56,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.94299999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.78100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302580,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302080,"kb_avail":94069260,"statfs":{"total":96636764160,"available":96326922240,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309329920},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Jan 15 18:50:39 2021","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":3.5369999999999999,"15min":2.9209999999999998},"min":{"1min":0.48499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.477,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":9.8089999999999993,"5min":4.0460000000000003,"15min":2.702},"min":{"1min":0.496,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":44.920999999999999,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.65100000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.82699999999999996}]}]},{"osd":0,"up_from":22,"seq":94489280662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":301951,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309197876},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:50:42 2021","interfaces":[{"interface":"back","average":{"1min":5.6719999999999997,"5min":6.5110000000000001,"15min":4.2320000000000002},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":37.043999999999997,"5min":50.948,"15min":50.948},"last":0.47499999999999998},{"interface":"front","average":{"1min":5.7919999999999998,"5min":5.7240000000000002,"15min":4.0140000000000002},"min":{"1min":0.53800000000000003,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":37.140000000000001,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.68999999999999995}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.753}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.82599999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293244,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":292735,"kb_avail":94078596,"statfs":{"total":96636764160,"available":96336482304,"internally_reserved":0,"allocated":512000,"data_stored":198102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":299760692},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.5949999999999998,"5min":5.1050000000000004,"15min":4.0620000000000003},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":38.216000000000001,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":3.5249999999999999,"5min":2.7149999999999999,"15min":2.3570000000000002},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.39500000000000002},"max":{"1min":19.937999999999999,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":7.8200000000000003,"5min":4.0890000000000004,"15min":3.1840000000000002},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":30.181999999999999,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":4.5679999999999996,"5min":2.9660000000000002,"15min":2.238},"min":{"1min":0.61499999999999999,"5min":0.51700000000000002,"15min":0.47899999999999998},"max":{"1min":23.802,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:50:52 2021","interfaces":[{"interface":"back","average":{"1min":3.677,"5min":3.024,"15min":3.5670000000000002},"min":{"1min":0.80600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":20.059000000000001,"5min":24.582999999999998,"15min":25.216000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":3.7719999999999998,"5min":3.1789999999999998,"15min":2.7869999999999999},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":20.16,"5min":24.696000000000002,"15min":24.696000000000002},"last":1.1870000000000001}]}]}]}} 2021-01-15T18:51:39.092 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.092 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2021-01-15T18:51:31.879512+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=31, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.092 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-01-15T18:51:30.828559+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=30, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.093 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.093 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.093 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.094 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.094 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.094 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.095 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-01-15T18:51:29.594666+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=29, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.095 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-01-15T18:51:31.556904+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=31, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.095 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.096 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-01-15T18:51:30.626149+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=30, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.096 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.096 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-01-15T18:51:32.601324+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=32, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.097 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-01-15T18:51:29.775771+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=29, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:39.097 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-01-15T18:51:59.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:51:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:51:59.367 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:51:59.378 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2021-01-15T18:51:58.709174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33432,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58225,"num_write_kb":232908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7200,"num_bytes_recovered":12763136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1200728,"kb_used_data":1976,"kb_used_omap":2,"kb_used_meta":1198717,"kb_avail":376286632,"statfs":{"total":386547056640,"available":385317511168,"internally_reserved":0,"allocated":2023424,"data_stored":767832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1227486364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4179,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9727,"num_write_kb":-38909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1438,"num_bytes_recovered":-1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002575"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12302","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:54.533102+0000","last_change":"2021-01-15T18:51:54.533261+0000","last_active":"2021-01-15T18:51:54.533102+0000","last_peered":"2021-01-15T18:51:54.533102+0000","last_clean":"2021-01-15T18:51:54.533102+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:54.533102+0000","last_undegraded":"2021-01-15T18:51:54.533102+0000","last_fullsized":"2021-01-15T18:51:54.533102+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9508","last_scrub_stamp":"2021-01-15T18:51:54.533097+0000","last_deep_scrub":"54'9508","last_deep_scrub_stamp":"2021-01-15T18:51:54.533097+0000","last_clean_scrub_stamp":"2021-01-15T18:51:54.533097+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12408","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:31.879513+0000","last_change":"2021-01-15T18:51:31.879612+0000","last_active":"2021-01-15T18:51:31.879513+0000","last_peered":"2021-01-15T18:51:31.879513+0000","last_clean":"2021-01-15T18:51:31.879513+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:51:31.879513+0000","last_undegraded":"2021-01-15T18:51:31.879513+0000","last_fullsized":"2021-01-15T18:51:31.879513+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9460","last_scrub_stamp":"2021-01-15T18:51:31.879512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:51:31.879512+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12447","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:30.828565+0000","last_change":"2021-01-15T18:51:30.828677+0000","last_active":"2021-01-15T18:51:30.828565+0000","last_peered":"2021-01-15T18:51:30.828565+0000","last_clean":"2021-01-15T18:51:30.828565+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:30.828565+0000","last_undegraded":"2021-01-15T18:51:30.828565+0000","last_fullsized":"2021-01-15T18:51:30.828565+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9492","last_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_deep_scrub":"54'9492","last_deep_scrub_stamp":"2021-01-15T18:51:30.828559+0000","last_clean_scrub_stamp":"2021-01-15T18:51:30.828559+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"12742","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:51.492011+0000","last_change":"2021-01-15T18:51:51.492170+0000","last_active":"2021-01-15T18:51:51.492011+0000","last_peered":"2021-01-15T18:51:51.492011+0000","last_clean":"2021-01-15T18:51:51.492011+0000","last_became_active":"2021-01-15T18:51:30.803006+0000","last_became_peered":"2021-01-15T18:51:30.803006+0000","last_unstale":"2021-01-15T18:51:51.492011+0000","last_undegraded":"2021-01-15T18:51:51.492011+0000","last_fullsized":"2021-01-15T18:51:51.492011+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9792","last_scrub_stamp":"2021-01-15T18:51:51.492007+0000","last_deep_scrub":"54'9792","last_deep_scrub_stamp":"2021-01-15T18:51:51.492007+0000","last_clean_scrub_stamp":"2021-01-15T18:51:51.492007+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10439","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796611+0000","last_change":"2021-01-15T18:41:52.787187+0000","last_active":"2021-01-15T18:51:29.796611+0000","last_peered":"2021-01-15T18:51:29.796611+0000","last_clean":"2021-01-15T18:51:29.796611+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:51:29.796611+0000","last_undegraded":"2021-01-15T18:51:29.796611+0000","last_fullsized":"2021-01-15T18:51:29.796611+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":39300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1505,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9756","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796708+0000","last_change":"2021-01-15T18:41:44.742834+0000","last_active":"2021-01-15T18:51:29.796708+0000","last_peered":"2021-01-15T18:51:29.796708+0000","last_clean":"2021-01-15T18:51:29.796708+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:51:29.796708+0000","last_undegraded":"2021-01-15T18:51:29.796708+0000","last_fullsized":"2021-01-15T18:51:29.796708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":38095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":466944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12605","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796628+0000","last_change":"2021-01-15T18:48:41.001576+0000","last_active":"2021-01-15T18:51:29.796628+0000","last_peered":"2021-01-15T18:51:29.796628+0000","last_clean":"2021-01-15T18:51:29.796628+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:51:29.796628+0000","last_undegraded":"2021-01-15T18:51:29.796628+0000","last_fullsized":"2021-01-15T18:51:29.796628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8358,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9722,"num_write_kb":38892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1396,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12873","reported_epoch":"59","state":"active+clean","last_fresh":"2021-01-15T18:51:29.796403+0000","last_change":"2021-01-15T18:48:56.925660+0000","last_active":"2021-01-15T18:51:29.796403+0000","last_peered":"2021-01-15T18:51:29.796403+0000","last_clean":"2021-01-15T18:51:29.796403+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:29.796403+0000","last_undegraded":"2021-01-15T18:51:29.796403+0000","last_fullsized":"2021-01-15T18:51:29.796403+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-15T18:39:25.483201+0000","last_clean_scrub_stamp":"2021-01-15T18:39:25.483201+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":39421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12687","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:47.952294+0000","last_change":"2021-01-15T18:51:47.952444+0000","last_active":"2021-01-15T18:51:47.952294+0000","last_peered":"2021-01-15T18:51:47.952294+0000","last_clean":"2021-01-15T18:51:47.952294+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:47.952294+0000","last_undegraded":"2021-01-15T18:51:47.952294+0000","last_fullsized":"2021-01-15T18:51:47.952294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9728","last_scrub_stamp":"2021-01-15T18:51:47.952290+0000","last_deep_scrub":"54'9728","last_deep_scrub_stamp":"2021-01-15T18:51:47.952290+0000","last_clean_scrub_stamp":"2021-01-15T18:51:47.952290+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12725","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:50.457940+0000","last_change":"2021-01-15T18:51:50.458133+0000","last_active":"2021-01-15T18:51:50.457940+0000","last_peered":"2021-01-15T18:51:50.457940+0000","last_clean":"2021-01-15T18:51:50.457940+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:50.457940+0000","last_undegraded":"2021-01-15T18:51:50.457940+0000","last_fullsized":"2021-01-15T18:51:50.457940+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9756","last_scrub_stamp":"2021-01-15T18:51:50.457931+0000","last_deep_scrub":"54'9756","last_deep_scrub_stamp":"2021-01-15T18:51:50.457931+0000","last_clean_scrub_stamp":"2021-01-15T18:51:50.457931+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12420","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:42.563022+0000","last_change":"2021-01-15T18:51:42.563169+0000","last_active":"2021-01-15T18:51:42.563022+0000","last_peered":"2021-01-15T18:51:42.563022+0000","last_clean":"2021-01-15T18:51:42.563022+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:42.563022+0000","last_undegraded":"2021-01-15T18:51:42.563022+0000","last_fullsized":"2021-01-15T18:51:42.563022+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9553","last_scrub_stamp":"2021-01-15T18:51:42.563017+0000","last_deep_scrub":"54'9553","last_deep_scrub_stamp":"2021-01-15T18:51:42.563017+0000","last_clean_scrub_stamp":"2021-01-15T18:51:42.563017+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12419","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:40.034518+0000","last_change":"2021-01-15T18:51:40.034637+0000","last_active":"2021-01-15T18:51:40.034518+0000","last_peered":"2021-01-15T18:51:40.034518+0000","last_clean":"2021-01-15T18:51:40.034518+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:40.034518+0000","last_undegraded":"2021-01-15T18:51:40.034518+0000","last_fullsized":"2021-01-15T18:51:40.034518+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9566","last_scrub_stamp":"2021-01-15T18:51:40.034513+0000","last_deep_scrub":"54'9566","last_deep_scrub_stamp":"2021-01-15T18:51:40.034513+0000","last_clean_scrub_stamp":"2021-01-15T18:51:40.034513+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12397","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:40.581075+0000","last_change":"2021-01-15T18:51:40.581198+0000","last_active":"2021-01-15T18:51:40.581075+0000","last_peered":"2021-01-15T18:51:40.581075+0000","last_clean":"2021-01-15T18:51:40.581075+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:40.581075+0000","last_undegraded":"2021-01-15T18:51:40.581075+0000","last_fullsized":"2021-01-15T18:51:40.581075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9474","last_scrub_stamp":"2021-01-15T18:51:40.581070+0000","last_deep_scrub":"54'9474","last_deep_scrub_stamp":"2021-01-15T18:51:40.581070+0000","last_clean_scrub_stamp":"2021-01-15T18:51:40.581070+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10135","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:45.026764+0000","last_change":"2021-01-15T18:51:45.026923+0000","last_active":"2021-01-15T18:51:45.026764+0000","last_peered":"2021-01-15T18:51:45.026764+0000","last_clean":"2021-01-15T18:51:45.026764+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:45.026764+0000","last_undegraded":"2021-01-15T18:51:45.026764+0000","last_fullsized":"2021-01-15T18:51:45.026764+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9885","last_scrub_stamp":"2021-01-15T18:51:45.026753+0000","last_deep_scrub":"54'9885","last_deep_scrub_stamp":"2021-01-15T18:51:45.026753+0000","last_clean_scrub_stamp":"2021-01-15T18:51:45.026753+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13122","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:48.465500+0000","last_change":"2021-01-15T18:51:48.465646+0000","last_active":"2021-01-15T18:51:48.465500+0000","last_peered":"2021-01-15T18:51:48.465500+0000","last_clean":"2021-01-15T18:51:48.465500+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:48.465500+0000","last_undegraded":"2021-01-15T18:51:48.465500+0000","last_fullsized":"2021-01-15T18:51:48.465500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'10040","last_scrub_stamp":"2021-01-15T18:51:48.465495+0000","last_deep_scrub":"54'10040","last_deep_scrub_stamp":"2021-01-15T18:51:48.465495+0000","last_clean_scrub_stamp":"2021-01-15T18:51:48.465495+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12431","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:39.885669+0000","last_change":"2021-01-15T18:51:39.885787+0000","last_active":"2021-01-15T18:51:39.885669+0000","last_peered":"2021-01-15T18:51:39.885669+0000","last_clean":"2021-01-15T18:51:39.885669+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:39.885669+0000","last_undegraded":"2021-01-15T18:51:39.885669+0000","last_fullsized":"2021-01-15T18:51:39.885669+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9538","last_scrub_stamp":"2021-01-15T18:51:39.885664+0000","last_deep_scrub":"54'9538","last_deep_scrub_stamp":"2021-01-15T18:51:39.885664+0000","last_clean_scrub_stamp":"2021-01-15T18:51:39.885664+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":33432,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58225,"num_write_kb":232908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7200,"num_bytes_recovered":12763136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215253,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298292,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297791,"kb_avail":94073548,"statfs":{"total":96636764160,"available":96331313152,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":304938036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.66600000000000004}]},{"osd":1,"last update":"Fri Jan 15 18:51:46 2021","interfaces":[{"interface":"back","average":{"1min":3.8519999999999999,"5min":5.0090000000000003,"15min":4.0990000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":31.707999999999998,"5min":31.707999999999998,"15min":31.707999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":4.6189999999999998,"5min":5.5190000000000001,"15min":3.8439999999999999},"min":{"1min":0.51200000000000001,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":29.690999999999999,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.85399999999999998}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.72699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302580,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302080,"kb_avail":94069260,"statfs":{"total":96636764160,"available":96326922240,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309329920},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":4.5750000000000002,"5min":2.6459999999999999,"15min":3.3599999999999999},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":11.710000000000001,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":2.286,"5min":2.1720000000000002,"15min":2.036},"min":{"1min":0.379,"5min":0.379,"15min":0.371},"max":{"1min":13.628,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.60699999999999998}]},{"osd":1,"last update":"Fri Jan 15 18:51:41 2021","interfaces":[{"interface":"back","average":{"1min":2.456,"5min":3.6869999999999998,"15min":2.9319999999999999},"min":{"1min":0.52400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.827999999999999,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.7509999999999999,"5min":4.4290000000000003,"15min":2.7480000000000002},"min":{"1min":0.48899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":18.498000000000001,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jan 15 18:51:02 2021","interfaces":[{"interface":"back","average":{"1min":10.932,"5min":5.2069999999999999,"15min":3.52},"min":{"1min":0.85299999999999998,"5min":0.47499999999999998,"15min":0.39200000000000002},"max":{"1min":32.691000000000003,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.747},{"interface":"front","average":{"1min":6.899,"5min":3.1379999999999999,"15min":2.8639999999999999},"min":{"1min":0.621,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":23.577999999999999,"5min":23.577999999999999,"15min":23.577999999999999},"last":0.81999999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302516,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302015,"kb_avail":94069324,"statfs":{"total":96636764160,"available":96326987776,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309263412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:51:43 2021","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":6.1399999999999997,"15min":4.2089999999999996},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":12.327999999999999,"5min":50.948,"15min":50.948},"last":0.58599999999999997},{"interface":"front","average":{"1min":2.7530000000000001,"5min":5.6790000000000003,"15min":4.0579999999999998},"min":{"1min":0.52600000000000002,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":12.419,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.77400000000000002}]},{"osd":2,"last update":"Fri Jan 15 18:51:03 2021","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":3.516,"15min":3.0150000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":6.5780000000000003,"5min":25.239999999999998,"15min":27.77},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.395,"5min":2.4169999999999998,"15min":2.6499999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":7.9630000000000001,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.79900000000000004}]},{"osd":3,"last update":"Fri Jan 15 18:51:09 2021","interfaces":[{"interface":"back","average":{"1min":3.7370000000000001,"5min":4.218,"15min":4.157},"min":{"1min":0.53700000000000003,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":16.359000000000002,"5min":56.615000000000002,"15min":56.615000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":2.746,"5min":3.169,"15min":3.3740000000000001},"min":{"1min":0.57099999999999995,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":16.472999999999999,"5min":24.734999999999999,"15min":56.927},"last":0.94499999999999995}]}]},{"osd":1,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297340,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":296831,"kb_avail":94074500,"statfs":{"total":96636764160,"available":96332288000,"internally_reserved":0,"allocated":512000,"data_stored":198102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":303954996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":4.7240000000000002,"15min":4.0149999999999997},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":10.965,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":2.7890000000000001,"15min":2.3370000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.39500000000000002},"max":{"1min":12.176,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":4.3380000000000001,"15min":3.1920000000000002},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":28.584,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":2.0350000000000001,"5min":3.0779999999999998,"15min":2.2639999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47899999999999998},"max":{"1min":12.361000000000001,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.57899999999999996}]},{"osd":3,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":2.9340000000000002,"15min":3.3969999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":11.087,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":3.0779999999999998,"15min":2.7280000000000002},"min":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":11.180999999999999,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.80100000000000005}]}]}]}} 2021-01-15T18:51:59.379 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2021-01-15T18:51:31.879512+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=31, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.379 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-01-15T18:51:30.828559+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=30, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.380 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.380 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.380 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.381 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2021-01-15T18:39:25.483201+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=39, tm_sec=25, tm_wday=4, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=15, tm_isdst=0) 2021-01-15T18:51:59.381 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-01-15T18:52:19.382 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-01-15T18:52:19.735 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-01-15T18:52:19.737 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2021-01-15T18:52:19.747 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2021-01-15T18:52:18.713698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19300,"num_write_kb":77200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1200728,"kb_used_data":1976,"kb_used_omap":2,"kb_used_meta":1198717,"kb_avail":376286632,"statfs":{"total":386547056640,"available":385317511168,"internally_reserved":0,"allocated":2023424,"data_stored":767832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2916,"internal_metadata":1227486364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16716,"num_objects":-4,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-19348,"num_write_kb":-77395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1564,"num_bytes_recovered":-12763136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002814"},"pg_stats":[{"pgid":"1.f","version":"54'9508","reported_seq":"12302","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:54.533102+0000","last_change":"2021-01-15T18:51:54.533261+0000","last_active":"2021-01-15T18:51:54.533102+0000","last_peered":"2021-01-15T18:51:54.533102+0000","last_clean":"2021-01-15T18:51:54.533102+0000","last_became_active":"2021-01-15T18:51:22.938492+0000","last_became_peered":"2021-01-15T18:51:22.938492+0000","last_unstale":"2021-01-15T18:51:54.533102+0000","last_undegraded":"2021-01-15T18:51:54.533102+0000","last_fullsized":"2021-01-15T18:51:54.533102+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9508","last_scrub_stamp":"2021-01-15T18:51:54.533097+0000","last_deep_scrub":"54'9508","last_deep_scrub_stamp":"2021-01-15T18:51:54.533097+0000","last_clean_scrub_stamp":"2021-01-15T18:51:54.533097+0000","log_size":9508,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9508,"num_write_kb":38032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"54'9460","reported_seq":"12412","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:52:10.921238+0000","last_change":"2021-01-15T18:52:10.921406+0000","last_active":"2021-01-15T18:52:10.921238+0000","last_peered":"2021-01-15T18:52:10.921238+0000","last_clean":"2021-01-15T18:52:10.921238+0000","last_became_active":"2021-01-15T18:41:21.811536+0000","last_became_peered":"2021-01-15T18:41:21.811536+0000","last_unstale":"2021-01-15T18:52:10.921238+0000","last_undegraded":"2021-01-15T18:52:10.921238+0000","last_fullsized":"2021-01-15T18:52:10.921238+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9460","last_scrub_stamp":"2021-01-15T18:52:10.921234+0000","last_deep_scrub":"54'9460","last_deep_scrub_stamp":"2021-01-15T18:52:10.921234+0000","last_clean_scrub_stamp":"2021-01-15T18:52:10.921234+0000","log_size":9460,"ondisk_log_size":9460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'9492","reported_seq":"12451","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:55.870141+0000","last_change":"2021-01-15T18:51:55.870258+0000","last_active":"2021-01-15T18:51:55.870141+0000","last_peered":"2021-01-15T18:51:55.870141+0000","last_clean":"2021-01-15T18:51:55.870141+0000","last_became_active":"2021-01-15T18:46:30.710003+0000","last_became_peered":"2021-01-15T18:46:30.710003+0000","last_unstale":"2021-01-15T18:51:55.870141+0000","last_undegraded":"2021-01-15T18:51:55.870141+0000","last_fullsized":"2021-01-15T18:51:55.870141+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9492","last_scrub_stamp":"2021-01-15T18:51:55.870136+0000","last_deep_scrub":"54'9492","last_deep_scrub_stamp":"2021-01-15T18:51:55.870136+0000","last_clean_scrub_stamp":"2021-01-15T18:51:55.870136+0000","log_size":9492,"ondisk_log_size":9492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"54'9792","reported_seq":"12742","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:51.492011+0000","last_change":"2021-01-15T18:51:51.492170+0000","last_active":"2021-01-15T18:51:51.492011+0000","last_peered":"2021-01-15T18:51:51.492011+0000","last_clean":"2021-01-15T18:51:51.492011+0000","last_became_active":"2021-01-15T18:51:30.803006+0000","last_became_peered":"2021-01-15T18:51:30.803006+0000","last_unstale":"2021-01-15T18:51:51.492011+0000","last_undegraded":"2021-01-15T18:51:51.492011+0000","last_fullsized":"2021-01-15T18:51:51.492011+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9792","last_scrub_stamp":"2021-01-15T18:51:51.492007+0000","last_deep_scrub":"54'9792","last_deep_scrub_stamp":"2021-01-15T18:51:51.492007+0000","last_clean_scrub_stamp":"2021-01-15T18:51:51.492007+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'9827","reported_seq":"10443","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:52:07.949685+0000","last_change":"2021-01-15T18:52:07.949845+0000","last_active":"2021-01-15T18:52:07.949685+0000","last_peered":"2021-01-15T18:52:07.949685+0000","last_clean":"2021-01-15T18:52:07.949685+0000","last_became_active":"2021-01-15T18:41:45.554867+0000","last_became_peered":"2021-01-15T18:41:45.554867+0000","last_unstale":"2021-01-15T18:52:07.949685+0000","last_undegraded":"2021-01-15T18:52:07.949685+0000","last_fullsized":"2021-01-15T18:52:07.949685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9827","last_scrub_stamp":"2021-01-15T18:52:07.949679+0000","last_deep_scrub":"54'9827","last_deep_scrub_stamp":"2021-01-15T18:52:07.949679+0000","last_clean_scrub_stamp":"2021-01-15T18:52:07.949679+0000","log_size":9827,"ondisk_log_size":9827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'9525","reported_seq":"9760","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:52:06.945891+0000","last_change":"2021-01-15T18:52:06.946033+0000","last_active":"2021-01-15T18:52:06.945891+0000","last_peered":"2021-01-15T18:52:06.945891+0000","last_clean":"2021-01-15T18:52:06.945891+0000","last_became_active":"2021-01-15T18:41:36.537082+0000","last_became_peered":"2021-01-15T18:41:36.537082+0000","last_unstale":"2021-01-15T18:52:06.945891+0000","last_undegraded":"2021-01-15T18:52:06.945891+0000","last_fullsized":"2021-01-15T18:52:06.945891+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9525","last_scrub_stamp":"2021-01-15T18:52:06.945885+0000","last_deep_scrub":"54'9525","last_deep_scrub_stamp":"2021-01-15T18:52:06.945885+0000","last_clean_scrub_stamp":"2021-01-15T18:52:06.945885+0000","log_size":9525,"ondisk_log_size":9525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"54'9724","reported_seq":"12609","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:52:01.980179+0000","last_change":"2021-01-15T18:52:01.980313+0000","last_active":"2021-01-15T18:52:01.980179+0000","last_peered":"2021-01-15T18:52:01.980179+0000","last_clean":"2021-01-15T18:52:01.980179+0000","last_became_active":"2021-01-15T18:48:28.049629+0000","last_became_peered":"2021-01-15T18:48:28.049629+0000","last_unstale":"2021-01-15T18:52:01.980179+0000","last_undegraded":"2021-01-15T18:52:01.980179+0000","last_fullsized":"2021-01-15T18:52:01.980179+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9724","last_scrub_stamp":"2021-01-15T18:52:01.980174+0000","last_deep_scrub":"54'9724","last_deep_scrub_stamp":"2021-01-15T18:52:01.980174+0000","last_clean_scrub_stamp":"2021-01-15T18:52:01.980174+0000","log_size":9724,"ondisk_log_size":9724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'9856","reported_seq":"12877","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:58.962476+0000","last_change":"2021-01-15T18:51:58.962595+0000","last_active":"2021-01-15T18:51:58.962476+0000","last_peered":"2021-01-15T18:51:58.962476+0000","last_clean":"2021-01-15T18:51:58.962476+0000","last_became_active":"2021-01-15T18:48:48.814857+0000","last_became_peered":"2021-01-15T18:48:48.814857+0000","last_unstale":"2021-01-15T18:51:58.962476+0000","last_undegraded":"2021-01-15T18:51:58.962476+0000","last_fullsized":"2021-01-15T18:51:58.962476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9856","last_scrub_stamp":"2021-01-15T18:51:58.962471+0000","last_deep_scrub":"54'9856","last_deep_scrub_stamp":"2021-01-15T18:51:58.962471+0000","last_clean_scrub_stamp":"2021-01-15T18:51:58.962471+0000","log_size":9856,"ondisk_log_size":9856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'9728","reported_seq":"12687","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:47.952294+0000","last_change":"2021-01-15T18:51:47.952444+0000","last_active":"2021-01-15T18:51:47.952294+0000","last_peered":"2021-01-15T18:51:47.952294+0000","last_clean":"2021-01-15T18:51:47.952294+0000","last_became_active":"2021-01-15T18:45:54.676580+0000","last_became_peered":"2021-01-15T18:45:54.676580+0000","last_unstale":"2021-01-15T18:51:47.952294+0000","last_undegraded":"2021-01-15T18:51:47.952294+0000","last_fullsized":"2021-01-15T18:51:47.952294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9728","last_scrub_stamp":"2021-01-15T18:51:47.952290+0000","last_deep_scrub":"54'9728","last_deep_scrub_stamp":"2021-01-15T18:51:47.952290+0000","last_clean_scrub_stamp":"2021-01-15T18:51:47.952290+0000","log_size":9728,"ondisk_log_size":9728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'9756","reported_seq":"12725","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:50.457940+0000","last_change":"2021-01-15T18:51:50.458133+0000","last_active":"2021-01-15T18:51:50.457940+0000","last_peered":"2021-01-15T18:51:50.457940+0000","last_clean":"2021-01-15T18:51:50.457940+0000","last_became_active":"2021-01-15T18:42:01.663262+0000","last_became_peered":"2021-01-15T18:42:01.663262+0000","last_unstale":"2021-01-15T18:51:50.457940+0000","last_undegraded":"2021-01-15T18:51:50.457940+0000","last_fullsized":"2021-01-15T18:51:50.457940+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9756","last_scrub_stamp":"2021-01-15T18:51:50.457931+0000","last_deep_scrub":"54'9756","last_deep_scrub_stamp":"2021-01-15T18:51:50.457931+0000","last_clean_scrub_stamp":"2021-01-15T18:51:50.457931+0000","log_size":9756,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'9553","reported_seq":"12420","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:42.563022+0000","last_change":"2021-01-15T18:51:42.563169+0000","last_active":"2021-01-15T18:51:42.563022+0000","last_peered":"2021-01-15T18:51:42.563022+0000","last_clean":"2021-01-15T18:51:42.563022+0000","last_became_active":"2021-01-15T18:49:50.005522+0000","last_became_peered":"2021-01-15T18:49:50.005522+0000","last_unstale":"2021-01-15T18:51:42.563022+0000","last_undegraded":"2021-01-15T18:51:42.563022+0000","last_fullsized":"2021-01-15T18:51:42.563022+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9553","last_scrub_stamp":"2021-01-15T18:51:42.563017+0000","last_deep_scrub":"54'9553","last_deep_scrub_stamp":"2021-01-15T18:51:42.563017+0000","last_clean_scrub_stamp":"2021-01-15T18:51:42.563017+0000","log_size":9553,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"54'9566","reported_seq":"12419","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:40.034518+0000","last_change":"2021-01-15T18:51:40.034637+0000","last_active":"2021-01-15T18:51:40.034518+0000","last_peered":"2021-01-15T18:51:40.034518+0000","last_clean":"2021-01-15T18:51:40.034518+0000","last_became_active":"2021-01-15T18:45:34.838514+0000","last_became_peered":"2021-01-15T18:45:34.838514+0000","last_unstale":"2021-01-15T18:51:40.034518+0000","last_undegraded":"2021-01-15T18:51:40.034518+0000","last_fullsized":"2021-01-15T18:51:40.034518+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9566","last_scrub_stamp":"2021-01-15T18:51:40.034513+0000","last_deep_scrub":"54'9566","last_deep_scrub_stamp":"2021-01-15T18:51:40.034513+0000","last_clean_scrub_stamp":"2021-01-15T18:51:40.034513+0000","log_size":9566,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'9474","reported_seq":"12397","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:40.581075+0000","last_change":"2021-01-15T18:51:40.581198+0000","last_active":"2021-01-15T18:51:40.581075+0000","last_peered":"2021-01-15T18:51:40.581075+0000","last_clean":"2021-01-15T18:51:40.581075+0000","last_became_active":"2021-01-15T18:43:09.564817+0000","last_became_peered":"2021-01-15T18:43:09.564817+0000","last_unstale":"2021-01-15T18:51:40.581075+0000","last_undegraded":"2021-01-15T18:51:40.581075+0000","last_fullsized":"2021-01-15T18:51:40.581075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9474","last_scrub_stamp":"2021-01-15T18:51:40.581070+0000","last_deep_scrub":"54'9474","last_deep_scrub_stamp":"2021-01-15T18:51:40.581070+0000","last_clean_scrub_stamp":"2021-01-15T18:51:40.581070+0000","log_size":9474,"ondisk_log_size":9474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'9885","reported_seq":"10135","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:45.026764+0000","last_change":"2021-01-15T18:51:45.026923+0000","last_active":"2021-01-15T18:51:45.026764+0000","last_peered":"2021-01-15T18:51:45.026764+0000","last_clean":"2021-01-15T18:51:45.026764+0000","last_became_active":"2021-01-15T18:46:07.705856+0000","last_became_peered":"2021-01-15T18:46:07.705856+0000","last_unstale":"2021-01-15T18:51:45.026764+0000","last_undegraded":"2021-01-15T18:51:45.026764+0000","last_fullsized":"2021-01-15T18:51:45.026764+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9885","last_scrub_stamp":"2021-01-15T18:51:45.026753+0000","last_deep_scrub":"54'9885","last_deep_scrub_stamp":"2021-01-15T18:51:45.026753+0000","last_clean_scrub_stamp":"2021-01-15T18:51:45.026753+0000","log_size":9885,"ondisk_log_size":9885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"54'10040","reported_seq":"13122","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:48.465500+0000","last_change":"2021-01-15T18:51:48.465646+0000","last_active":"2021-01-15T18:51:48.465500+0000","last_peered":"2021-01-15T18:51:48.465500+0000","last_clean":"2021-01-15T18:51:48.465500+0000","last_became_active":"2021-01-15T18:51:14.903837+0000","last_became_peered":"2021-01-15T18:51:14.903837+0000","last_unstale":"2021-01-15T18:51:48.465500+0000","last_undegraded":"2021-01-15T18:51:48.465500+0000","last_fullsized":"2021-01-15T18:51:48.465500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'10040","last_scrub_stamp":"2021-01-15T18:51:48.465495+0000","last_deep_scrub":"54'10040","last_deep_scrub_stamp":"2021-01-15T18:51:48.465495+0000","last_clean_scrub_stamp":"2021-01-15T18:51:48.465495+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'9538","reported_seq":"12431","reported_epoch":"60","state":"active+clean","last_fresh":"2021-01-15T18:51:39.885669+0000","last_change":"2021-01-15T18:51:39.885787+0000","last_active":"2021-01-15T18:51:39.885669+0000","last_peered":"2021-01-15T18:51:39.885669+0000","last_clean":"2021-01-15T18:51:39.885669+0000","last_became_active":"2021-01-15T18:44:24.711904+0000","last_became_peered":"2021-01-15T18:44:24.711904+0000","last_unstale":"2021-01-15T18:51:39.885669+0000","last_undegraded":"2021-01-15T18:51:39.885669+0000","last_fullsized":"2021-01-15T18:51:39.885669+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'9538","last_scrub_stamp":"2021-01-15T18:51:39.885664+0000","last_deep_scrub":"54'9538","last_deep_scrub_stamp":"2021-01-15T18:51:39.885664+0000","last_clean_scrub_stamp":"2021-01-15T18:51:39.885664+0000","log_size":9538,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":4179,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19300,"num_write_kb":77200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154724,"ondisk_log_size":154724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":24,"seq":103079215257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298292,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297791,"kb_avail":94073548,"statfs":{"total":96636764160,"available":96331313152,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":304938036},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.7829999999999999,"15min":3.427},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":10.539,"5min":38.780999999999999,"15min":38.780999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.677,"5min":2.5009999999999999,"15min":3.0379999999999998},"min":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":10.909000000000001,"5min":42.018999999999998,"15min":42.018999999999998},"last":0.70099999999999996}]},{"osd":1,"last update":"Fri Jan 15 18:51:46 2021","interfaces":[{"interface":"back","average":{"1min":3.8519999999999999,"5min":5.0090000000000003,"15min":4.0990000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":31.707999999999998,"5min":31.707999999999998,"15min":31.707999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":4.6189999999999998,"5min":5.5190000000000001,"15min":3.8439999999999999},"min":{"1min":0.51200000000000001,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":29.690999999999999,"5min":38.700000000000003,"15min":38.700000000000003},"last":0.88600000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:51:17 2021","interfaces":[{"interface":"back","average":{"1min":5.218,"5min":4.8630000000000004,"15min":3.464},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.39600000000000002},"max":{"1min":26.478999999999999,"5min":42.475000000000001,"15min":42.475000000000001},"last":0.745},{"interface":"front","average":{"1min":3.177,"5min":4.258,"15min":3.7850000000000001},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.503},"max":{"1min":9.5690000000000008,"5min":25.233000000000001,"15min":25.233000000000001},"last":0.83399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302580,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302080,"kb_avail":94069260,"statfs":{"total":96636764160,"available":96326922240,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":309329920},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:52:05 2021","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":2.4470000000000001,"15min":3.1579999999999999},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.33800000000000002},"max":{"1min":4.0599999999999996,"5min":16.923999999999999,"15min":36.380000000000003},"last":0.88},{"interface":"front","average":{"1min":0.84999999999999998,"5min":2.0409999999999999,"15min":2.0270000000000001},"min":{"1min":0.434,"5min":0.379,"15min":0.371},"max":{"1min":5.0629999999999997,"5min":27.861999999999998,"15min":27.861999999999998},"last":0.94499999999999995}]},{"osd":1,"last update":"Fri Jan 15 18:51:41 2021","interfaces":[{"interface":"back","average":{"1min":2.456,"5min":3.6869999999999998,"15min":2.9319999999999999},"min":{"1min":0.52400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":11.827999999999999,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":4.4290000000000003,"15min":2.7480000000000002},"min":{"1min":0.48899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":18.498000000000001,"5min":44.920999999999999,"15min":44.920999999999999},"last":0.97699999999999998}]},{"osd":3,"last update":"Fri Jan 15 18:52:05 2021","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":4.5979999999999999,"15min":3.448},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.39200000000000002},"max":{"1min":10.053000000000001,"5min":32.691000000000003,"15min":32.691000000000003},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.2989999999999999,"5min":3.0590000000000002,"15min":2.7989999999999999},"min":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":10.156000000000001,"5min":23.577999999999999,"15min":23.577999999999999},"last":1.069}]}]},{"osd":0,"up_from":22,"seq":94489280670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302516,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":302015,"kb_avail":94069324,"statfs":{"total":96636764160,"available":96326987776,"internally_reserved":0,"allocated":503808,"data_stored":189910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":309263412},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 15 18:51:43 2021","interfaces":[{"interface":"back","average":{"1min":2.5920000000000001,"5min":6.1399999999999997,"15min":4.2089999999999996},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":12.327999999999999,"5min":50.948,"15min":50.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.7530000000000001,"5min":5.6790000000000003,"15min":4.0579999999999998},"min":{"1min":0.52600000000000002,"5min":0.38800000000000001,"15min":0.375},"max":{"1min":12.419,"5min":50.031999999999996,"15min":50.031999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Fri Jan 15 18:52:06 2021","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":3.1179999999999999,"15min":2.9489999999999998},"min":{"1min":0.44700000000000001,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":12.615,"5min":25.239999999999998,"15min":27.77},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.9279999999999999,"15min":2.5419999999999998},"min":{"1min":0.54700000000000004,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":1.163,"5min":21.614999999999998,"15min":22.765000000000001},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Jan 15 18:52:11 2021","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":2.9689999999999999,"15min":3.8919999999999999},"min":{"1min":0.52000000000000002,"5min":0.49299999999999999,"15min":0.41699999999999998},"max":{"1min":0.92100000000000004,"5min":16.359000000000002,"15min":56.615000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":2.0550000000000002,"15min":3.2869999999999999},"min":{"1min":0.59899999999999998,"5min":0.48699999999999999,"15min":0.48199999999999998},"max":{"1min":0.98999999999999999,"5min":17.001999999999999,"15min":56.927},"last":0.82899999999999996}]}]},{"osd":1,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297340,"kb_used_data":500,"kb_used_omap":0,"kb_used_meta":296831,"kb_avail":94074500,"statfs":{"total":96636764160,"available":96332288000,"internally_reserved":0,"allocated":512000,"data_stored":198102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":303954996},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":4.7240000000000002,"15min":4.0149999999999997},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":10.965,"5min":50.558999999999997,"15min":50.558999999999997},"last":0.998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":2.7890000000000001,"15min":2.3370000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.39500000000000002},"max":{"1min":12.176,"5min":49.470999999999997,"15min":49.470999999999997},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":4.3380000000000001,"15min":3.1920000000000002},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":28.584,"5min":30.181999999999999,"15min":30.181999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.0350000000000001,"5min":3.0779999999999998,"15min":2.2639999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47899999999999998},"max":{"1min":12.361000000000001,"5min":24.795000000000002,"15min":24.795000000000002},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Jan 15 18:51:52 2021","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":2.9340000000000002,"15min":3.3969999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":11.087,"5min":24.582999999999998,"15min":25.216000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.7370000000000001,"5min":3.0779999999999998,"15min":2.7280000000000002},"min":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.54100000000000004},"max":{"1min":11.180999999999999,"5min":24.696000000000002,"15min":24.696000000000002},"last":0.79900000000000004}]}]}]}} 2021-01-15T18:52:19.750 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2021-01-15T18:52:20.040 INFO:teuthology.misc:Shutting down mds daemons... 2021-01-15T18:52:20.041 INFO:teuthology.misc:Shutting down osd daemons... 2021-01-15T18:52:20.041 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-01-15T18:52:20.041 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.120 INFO:tasks.ceph.osd.0:Stopped 2021-01-15T18:52:20.121 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-01-15T18:52:20.121 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.193 INFO:tasks.ceph.osd.1:Stopped 2021-01-15T18:52:20.194 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-01-15T18:52:20.194 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.272 INFO:tasks.ceph.osd.2:Stopped 2021-01-15T18:52:20.273 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-01-15T18:52:20.273 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.359 INFO:tasks.ceph.osd.3:Stopped 2021-01-15T18:52:20.360 INFO:teuthology.misc:Shutting down mgr daemons... 2021-01-15T18:52:20.360 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2021-01-15T18:52:20.361 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.378 INFO:tasks.ceph.mgr.x:Stopped 2021-01-15T18:52:20.379 INFO:teuthology.misc:Shutting down mon daemons... 2021-01-15T18:52:20.379 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-01-15T18:52:20.380 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.430 INFO:tasks.ceph.mon.a:Stopped 2021-01-15T18:52:20.430 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-01-15T18:52:20.431 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.480 INFO:tasks.ceph.mon.b:Stopped 2021-01-15T18:52:20.481 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-01-15T18:52:20.481 INFO:teuthology.orchestra.run:waiting for 300 2021-01-15T18:52:20.531 INFO:tasks.ceph.mon.c:Stopped 2021-01-15T18:52:20.532 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-15T18:52:20.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-01-15T18:52:20.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:52:20.647 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2021-01-15T18:52:20.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:52:20.903 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2021-01-15T18:52:21.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:52:21.046 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-01-15T18:52:21.129 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi148.front.sepia.ceph.com 2021-01-15T18:52:21.129 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2021-01-15T18:52:21.193 INFO:tasks.ceph:Archiving mon data... 2021-01-15T18:52:21.194 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643/data/mon.a.tgz 2021-01-15T18:52:21.194 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2021-01-15T18:52:21.219 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.zKvNlQZatU 2021-01-15T18:52:21.219 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zKvNlQZatU 2021-01-15T18:52:21.668 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.zKvNlQZatU 2021-01-15T18:52:21.806 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.zKvNlQZatU is 2MB 2021-01-15T18:52:21.916 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.zKvNlQZatU 2021-01-15T18:52:21.930 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643/data/mon.b.tgz 2021-01-15T18:52:21.931 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2021-01-15T18:52:21.984 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.bmZ8zfePeq 2021-01-15T18:52:21.985 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bmZ8zfePeq 2021-01-15T18:52:22.371 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.bmZ8zfePeq 2021-01-15T18:52:22.451 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.bmZ8zfePeq is 2MB 2021-01-15T18:52:22.564 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.bmZ8zfePeq 2021-01-15T18:52:22.579 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643/data/mon.c.tgz 2021-01-15T18:52:22.579 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2021-01-15T18:52:22.632 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.YtxtUm8eJW 2021-01-15T18:52:22.633 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.YtxtUm8eJW 2021-01-15T18:52:22.998 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.YtxtUm8eJW 2021-01-15T18:52:23.080 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.YtxtUm8eJW is 1MB 2021-01-15T18:52:23.194 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.YtxtUm8eJW 2021-01-15T18:52:23.209 INFO:tasks.ceph:Cleaning ceph cluster... 2021-01-15T18:52:23.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-01-15T18:52:23.294 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-01-15T18:52:23.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-01-15T18:52:23.321 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-01-15T18:52:23.328 INFO:tasks.ceph:Archiving crash dumps... 2021-01-15T18:52:23.330 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643/remote/smithi148/crash 2021-01-15T18:52:23.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-01-15T18:52:23.393 INFO:tasks.ceph:Compressing logs... 2021-01-15T18:52:23.394 DEBUG:teuthology.orchestra.run.smithi148:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-01-15T19:10:51.544 INFO:tasks.ceph:Archiving logs... 2021-01-15T19:10:51.567 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788643/remote/smithi148/log 2021-01-15T19:10:51.569 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/log/ceph -- . 2021-01-15T19:11:47.783 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2021-01-15T19:11:47.787 DEBUG:teuthology.task.console_log:Killing console logger for smithi148 2021-01-15T19:11:47.788 DEBUG:teuthology.task.console_log:Killing console logger for smithi148 2021-01-15T19:11:47.789 DEBUG:teuthology.exit:Finished running handlers